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ABSTRICi 

Ihe valid interpretation of informtition related to 
tie ^outcomes of vocational ^education is a persistent and pcofound 
probleo of the profession. Some of the reasons for tbi;-^ problem are 
difficulties cf communication, problems of definition, problems of 
collecting asd interpreting data, and lack of documentation* The 
prisiary problem appears to be lack of comminly acLspxed definitiDns 
._^#-*HTOcrt±onal -Student f " "relatednass" of training to employ raent, 
^ job perf ormancer " and "job satisfaction," To address these concerns, 
a National Conference on outcome Measures tor Votiational Education 
was held in August 197S* The four major pa|fers presented at this 
conference are contained ^ in this document. In thi first paper, Mary 
Bach Klavit explains the values of the various guoups directly 
concernad with vocational education, analyses the impact that these 
values have on the perspectiva through which members of each group 
view vocational education, and ascertains the ii^plications for 

.choosing and interpretirig outcome measurei in v jcational education. 
Then John Jennings describes the context in which the new evaluation 
re g iilrament s f or'^oc^^±OTT3r^educa^om--ca mt^^ a 1 

Education Aoendments of 1976, and the nature of those requirements. 
In the third paper^ Donald Drewes exa lines tie effects of 
stanflardi2#d outcoffle measures through feieral, itate, and local 

' levels* Finally, Grant Venn suggests that traditional success 
criteria in both work preparation and in general education are not 
relavant to the future* somehow, the tw i must be put together and 
changed if vocational education is to tniat the needs of both leacner 
and society. (KC) 
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THE NATIONAL CENTER MISSION STATEMENT 



The Nitional Center for Reiearch In Vocational Education'i misiion is 
to increaia the ability of diverie agencies, institutions, and organizations 
to solve educational problems relating to individual career planning, 
preparation, and progression. The National Center fulfills its mission by: 

a Generating knowledge through resiarch 

• Developing educational programs and products 

• Evaluating individual program needi and outcomes 
® Installing educational programs i>nd products 

e Operating information systems and sen/ices 

• Conductrng leadership development and training 
programs 



The study reported, herein was performed pursuRnt to a contract (#400-77-0700) 
from the National Institute cf Education. The opinions expressed herein do not 
necessarily reflect the position or policy of the National institute of Education and 
no official endorsement by the National Institute of Education should be inferred. 

The National Center for Research in Vocational Education does not discriminate 
against any individual because of race, color, creed, or sex. 



FOREWORD 



Interpreting outcomes of vocationai education is a major concern of leachers. ^ 
administrators advisory councils, and policy makers for vocatipnal education at ail 
levels The concerns focus on vocational education's response to the requirements and 
Intent' of the Education Amendments of 1976 and to the need for effective vocational 
education programs and services to prepare individuals for employment. 

The National Institute of Education recognized the confusion surrounding the 
interoretation of current vocational education outcome data and contracted with the 
National Center for Research in Vocational Education to examine the issues concerning 
interpretability of outcome data. 

The National Center for Research in v^ocational Education is indebted to Grant Venn, 
Marv Kievit and Donald Drewes for their commissioned papers and John Jennings tor 
his address presented at the National Conference on Outcome Measures for Vocational 
Education held August 16-18, 1978, in Louisville, Kentucky. 

The papers were compiled into this single collection for your use. (They also appear as 
a pa-t of Interpreting Outcome Measures in Vocational Education; Final Report.) 
Toqether the papers present several ideas about collecting, analyzing, and usmg^ 
vocational education outcome data presented by prominent professionals m the field. 
The coilection provides a valuable overview of the issues surrounding vocational 
educatioh outcome data, 

Recoqnition for their efforts are due to many other persons including the projec; staff of 
Floyd McKinney, project director; Kenney Gray, research specialist and Mane Abram, 
qraduate research associate; the evaluation division staff including N. L, McCaslm. 
associate director and former acting project director; Jerry Walker, fo^rner associate 
director Stephen Franchak, research specialist; Janet Weiskott, graduate research 
associate- Beth Harvey, qraduate research associate and other staff of the National 
^er for Research in Vocational Education, Significant contributions to the conduct of 
the project were also made by Robert Stump, project officer, the National Institute of 
Education; and the national advisory committee for the project, 

Robert E, Taylor 

Executive Director 

The National Center for Research 

in Vocationai Education 
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iNTRODUCTION 



Problem 



The valid interpretation of information related to the outcomes of vocational education 
ii, a persistent and profound problem of the profession. Congressional sta^ merr^.. 
the National Advisory Council for Vocational Education and num^erous other agencu.s 
have expressed concern about the interpretability of vocational education outcome data. 

There are several reasons for this concern. For example, various reports ha-- indic^ated 
problems of definition; problems of communication; and problems of collet ng and 
Hterpreting data. Also, there is a lack of documentation concernmg the processes used 

collect outcome data. Although this list of reasons is not inclusive, it does appear that 
the primary problem is definitional. Specifically, there is a lack cr commonly accepted 
standard definitions of; 

• wooationa! student 

• relatedness of training to employment 

• job performance (employers' view of former vocational students) 

. job satisfaction (former students' views of occupational and other life roles). 

For " ocationa! student" and "relatedness" the major definitional (and subsequent 
interpretability) issues stem from seemingly inconsistent and mcomplete uses of the 
terms by state and local education agencies. The problems with 'job performance ..nd 
"job satisfaction" stem not so much from inconsistent usage as from a lack oi 
measurement tools and processes by which one could even begin to grasp their 
operational meaning, 

Interpreling Outrome Measurts for 
Vocational Education Project 

Recognizing the types of concerns surrounding the interpretation of vocational 
education outcome data, the National Institute of Education contracted with the 
^Sial Center for Research in Vocational Education to examine the ^e^P^^^b^" ^ °f 
he available outcome data and to examine other outcome measures. Specifically, the 
problem of int^^rpreting vocational education outcome data was addressed through two 



ihe inconsistent usage of the "voc; lal student" and 

,u,utedness" among state and local vocational education agencies. 

To determine how different definitions of "vocational student" and Telatedness" 
might affect the interpretability of vocational education impact data, 



Project Activities 



In order to undertake the project objectives and disseminate the findings, the project 
was divided into four primary activities. 

p Case -Study. A case study was conducted to identify and describe the factors 
associated with the differences in job placement rates in field trained and related 
fields as reported by states. The issue areas considered in the case study were (I) 
background, (2) definitions of key terms, (3) process of collecting placement 
data, (4) utilization of placement data, and (5) placement function. 

# Commlssiontd Papers. Three topics were selected to be dealt with in the form of 
commissioned papers. The topics and the authors were: 

L Perspectivism in Choosing and Interpreting Outcome Measures in Vocational 
Education— Mary Bach Kievit 

2. Outcome Standardization for Compliance or Direction: The Crit nal 
Distinction— Donald W, Drewes 

3. Criteria Against Which Vocational Education Should Be Held Accountable = 
Grant Venn 

# Vocational iducatlon Measures: Instruments to Survey Former Students and 
Their Employers. A handbook of instruments to measure (i) job satisfaction, (2) 
job performance, (3) former students' perceptions of vocational training, and (4) 
employers' perceptions of students' vocational training was prepared. The 
handbook was designed to assist the practitioner by providing abstracts of 
instruments, copies of instruments, and a suggested method of choosing an 
instrument that fits the practitioner's purpose. It also describes the concepts and 
complexities of measuring the four categories of instruments included in the 
handbook. 

® National Conference, The National Conference on Outcome Measures for 
Vocational Education was held August IS-18, 1978 in Louisville, Kenuicky, the 
material in this publication is drawn from presentations made at that conference. 



Collection of Papers 

In an effort to disseminate as widely as possible the ideas and information about 
interpreting vocational education outnome measures obtrr J during the project, the 
four majc iparr j .nted at the .atlo. ... Jonfe^ence on Outcome Measures for 
Vocational education have been brought together into a single volume, Specificaily, the 
three commissioned papers and the address by John Jennings were selected f jr 
inclusion in the publication. The papers raise Important issues regarding the collr:iti^^^f 
analysis and use of vocationai education outcome d^tq it js hoped that the collectioii 
a whole will provide an overview from ^^/hi n ^he reaaur can select those ideas which 
can he Rdaptfd to mtet individual needa. 



PiRSPiCTfVISM IN CHOOSfNG AND iNTERPRETING 
OUTCOME MiASURES IN VOCATIONAL EDUCATION 



Mary Bach Kievit 
Purpose and Complexitits 



During the period of the late sixties and into the seventies, some writers have captured 
public attention and turned that attention to some of the negative outcomes of salient 
social and individual values. Future Shock, Zero Growth, by the Club of Rome corfe 
immediately to mind, as does a more recent volume Small is BeautifuL These books, 
among others, are manifestations of reexaminations of social directions, social values, 
and the implications of these for the present and the future. These reexaminations 
usually include some "pointing of fingers," attributing blame, and education at all levels 
has had its portion of these. There is a shared concern in many sectors for ansv^ering 
the question of "Hovv does what we produce, provide/ accomplish, match the needs, 
wants, expectations of our constituencies?*' In the educational arena the data to provide 
highly reliable and valid answers are sparse. The wisest answer from many vantage 
points is to focus on processes for seeking the kind of data which will provide these 
answers. Yet it is, nonetheless, valuable to ask, "What do we know?" and "What can we 
reasonably infer from what we knew?" What are the implications for action? 

The three-fold purpose of this paper is, to explicate as much as existing knowledge 
enables us to, the values of the various groups directly concerned with vocational 
ed'ication; to analyze the impact that these values have on the perspective through 
which members of each group view vocational education; and to ascertain the 
implications for choosing and interpreting outcoma measures in vocational education^ , 

The task as delimited sounds relatively simple and straight forward. In reality, it did not 
prove to be so. Delving Into diverse streams of literrture showed that there are many 
definitions of values. The Information available on values is very uneven in terms of the 
groups from which the data have been derived; the time period at which it has been 
obtained; and the direct relevancy to vocational education of th^ definitions of vali 
and hence, the data coh^cted. 

Some other nnn^^ nnr. jd from the multidimensionQlity of groups and ' idividuals. 

^ oh po^^ . js for ner/himself as an individual, and holds valuer relative to 

i ms each performs, e.g., as educator, but also as parent: as legislator 

concerned about the social welfare, but also as legislator concerned about reelection. In 
brief, there is a multrdimensionality about each person and about the . jncerns held by 
various groups that make definite statements hazardous, ac luidelines for action in this 
a, and compel us toward tentative statements to be tested for soundiiess in the 

/erse situations to which a concern for values has import. 

In an era touted for the rapidity of change, the question of stability anrf^Biinge must be 
confronted directly both in the longer historical coqtext and also withinj^ lifetime of 
individuals. While considering the question of stability or change, the^ nature of change 
an|J the sources of change come into play. Change can be said to occur when one 
moves from the diffuse, the general, to the differentiated, the specific. But change ui this 
nature does not preclude changing the degree of importance corded one value as it 
comes in conflict with another; or indeed of ceasing to value what had been valued. 



and should be varies among educators. Some hold that education shoula mcuicate 
specific values. Others hold that education provides the experiences and the substance 
from which individuals form values, in a highly individual way, so that education 
influences value formation only indirectly. Still others contend that the proper roie ot 
education is to directly faci itate the clarification of values, and some rationai 
examination of the implications of holding specific values. As one seeks tentative 
answers to these questions, one must make decis'ons about the nature of evidence 
which one wil! employ. For the literature provides evidence emanating from a deductive 
mode, the introspective, logical analytical. philosophicaL and the behavioral science 

In addition, acceptance of a specific definition of value, ana clusters of values, enables 
ans to use data as indicators of the presence or absence of valuing and m so doing to 
draw inferences. 

In each case, the delineation of the reality investigated, both through the definitions, 
and the subsequent operationalization into systematic measures, creates inherent 
limitations. Self-report measures, for example, raise the serious question as to the level 
of awareness of individuals regarding their values, the extent to which predetermined 
labels have, in fact, distorted the reality of values for specific Individuals and specific 
groups. Having shared with you some sense of issues to be resolved in the process of 
developing a position, let me now share v/iih you the results of that process. 



Conceptuaf Framework and Types of Evidence 

The conceptual framework within which I chose to-inquire is that of the behaviora! 
sciences. Sociologist Robin Williams is rather generally recognized as one of the miore 
profound thinkers among sociologists and has provided a thoughtful analysis of value 
orientations whfch exist in the American culture. Sociai-psychologist Milton Rokeach 
has built upon his own inquiry in beliefs and attitudes to study further the nature of 
values held by individuals, and the way in which individual values differ among persons 
based on inclusion in socioeconomic categories, occupational categories^^eacial and 
religious categories among others, Rokeach is knowledgeable of Wiiliams''vv(prk and 
draws upon and expands some aspects of It. Another approach closely linkedMS that of 
Abraham Maslow, the humanist psychologist who postulated a hierarchy of basic sets 
of human need. 

Relative to the types of evidence, greatest weight has been given to knowledge derived 
from empirical data through systematic research. Due to the limits of this source, other 
types of data have been utilized as a basis for drawing inferences. Some of these are in 
the category of informed opinion. Some are derived from testimony to legislative 
committees. Some are quite impressionistic based on personal experiences with parents 
and students through time. In brief, as a result of the significance of the subject. I have 
employed all of the information I had. Thus, in Spinozian terms, it ranges from hearsay, 
through vague experience, knowledge reached by reasoning, and by immediate 
deduction and direct perception, with an effort to use the last two as much possible. 



\ 



Values: V/hat Are Thsy? 
How Do We Discern Them? 



Values according to Williams, is "any aspect of a situation, event, or object that is 
invested with a preferential interest as being ■good,' 'bad.- 'desiraDle and 
the like " (WiMiams 1956- p. 374) Values have a conceptual element; they are arfectively 
charqed representing actual or potential emotional mobilization: they are not conr-ete 
goals of action but criteria by which goals are chosen; they are important, not tnvia! or 
of slight concern. (Williams, p. 374) 

Social values are reaarded as matters of collection welfare by an effective consensus of 
the qroup in sum, both for groups and individuals ^values are modes of organizing 
conduct-meaningful, affectively invested pattern principles that guide human action. 
(Williams, p 375). 

Values are viewed as constituting a type of continuum. Values concern the goals of 
action in addition to being components in the selection of means to acnieve the goals. 

Er^Dirically some of Jhe evidences of values are; the choices made; the directions of 
interest (that to which people pay attenticn); statements of what is valued, emotional 
reactions to statements, implicit premises^not stated because they are taken ,or 
granted, explanations and reasons given for conduct: rewards and punishments, i e 
social sanctions, (Williams, pp. 373-382) 

Williams differentiates between dominant and subordinate values noting that for a group 
or system as a whole these criteria are useful: 

' extensiveness-proportlon of population and activity manifesting the value, 

• the duration of the value— has it persisted through time, 

• the intensity with which the value is sought or maintained-=e.g„ promptness, 
certainty, and severity of sanctions when the value is threatened, and 

• the prestige of value carriers, persons, objects or organisations considered to be 
bearers of the value, (pp. 332-383) 

in a pluralistic society such as this, precise and detailed characterization of values can 
only be done for carefully delimited segments of society. In view of the need, however, 
for some core values for any society to have the integration essential to survival, major 
value orientations are apparent. Different groups within society may place the value 
components of such major orientations In different positions of dominance and 
subordination and may omit some, and add others, 



Major Value Orientations 

Utilizing the above criteria, Williams, in 1956 examined available evidence to specify 
major value orientations in American society. The validity of his analysis is attested to 
by the commonalities found with analyses completed in the I970's to address tine 
question of stability and/or change in values in America. These nnajor value orientations 



have import for t-is mauiry because the vaiues of the varin.is n'C.ir- .^-r 
vocationai educaiion emanate from this social=cuiiurai hase. 

Major value^orientation, as ideal types and Kinds of ceniral tenaencies ny wr-;cr to ■lOt-;? 
departures, accordmg to Williams are; 

L Acnievement-success: stresses personal achievement especiaMy secular cccura- 
tionai acnievement, and acquisition of generaiiy acceptec symbois of succv^ys 

2. Activity and work: mastery and domination of the externa! world with direciea 
and disciplined a.iivity in a regular occupation a particular forn' of the oasic 
orientation. 

3. Moral orientation: an ethical quality of a particular type that mciudes principles to 
work hard, lead an orderly life, to have a name for mtegnty and fair deaiirg. not 
to spend one's substance in reckless display, to have the resolution to carry out 
the purposes you undertake. 

4. Humanitarian mores: disinterested concern and helpfulness, mcluding gefso:]m 
kindUness, aid and comfort, and organized phi anthropy. 

5. Ffficiency and practicality: unites activity and substantive rationality, focusirig 
upon a choice of the lost effective means for a given end. Manifestation the 
appreciation of technical values in skilled trades, technical, quasi-profes rial 
and professional vocations with systematic indoctrination in the standaras of 
"doing a good job." "Practicality as to concrete goals of action corresponaingty 
has meant the canalyzihg of action in the service of those specific life niodels 
most highly approved in the general culture--broadly speaking, rational 
strenuous, competitive striving for personal validation through occupational 
success" (p. 403). 

6. Progress: emphasis on the future rather than the present or past, receptivity to 
change, faith in the perfectibility of the common man—optimism. 

7. Material comfort: high level of material comifort is sought after; standard of living 
has its attached meanings however for symbols of success, competence, and 
power. 

8. Equality: a value complex subject to much strain includes equality in 
interpersonal ^'elations as a goal and standard— with compromises in practice: 
equality of specific formal rights as in the strong and continuing strain for. 
equality for legal rights for all citizens; equality of opportunity to economic 
freedom and individual achievement rather than equality of condition in either 
economics or achievement. 

9. Freedom: for the individual as an integral agent relatively autonomous and 
morally responsible. The corollary is that a great variety of forms of personal 
dependencies constitute a loss of freedom. For example, "freedom or thought so 
that the truth may prevail; freedom of occupation so that careers may be open to 
talent; freedom of self-government, so that no one may be compelled against his 
will" (Becker-^Williams^ p. 434). 
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id External conformity: in a very heterogeneous culture conformity in externals 
helps to make it possible to continue the society in spite of many classes of 
Interests and basic values. 

11 Science and secular rationality: an emphasis reflecting the rationalistic-- 
individualistic tradition; i.e., disciplined, rational, functional, active, requiring 
systematic diligence and honesty. It Is compatible with strivings for mastery of 
the environment, denials of frustrations and refusal.? to accept the Idea of a 

" fundamentally unreasonable and capricious world. 

12. Nationalism— patriotism: widespread satisfaction of people with tbn country. 

13 Democracy: based on the impiicit belief in natural law as opposed to personal 
rule and in the moral autonomy of the individual with a theme of democracy as a 

■ procedure In distributing power and settling conflicts. 

14 Individual personality: the valuing of the development of individual personality to 
the end that the person is independent, responsible, and self-respecting and thus 
worthy of concern and resp^ect in one's own right; in sum, valuing a certam kind 
of individual. 

15 Racism and group superiority: the ascription of value and privilege to individuals 
on the basis of race or particularistic group membership according to birth in a 
particular ethnic group," social class or related social category, A pervasive and 
powerful counter-current to the values of equality, humanitarian values, political 
freedoms. 



Change or Stability? 

■WllllarTis formulated these major value orientations in 1956. Have these persisted? Have 
these changed? Alex Inkeles (Change, 1977. p. 25) has examined American perceptions 
to look for continuities and discontinuities with the past. A comparispn of the areas he 
treats as "perceptions" with the major value orientations of Williams shows a number of 
commonalltiei. Inkeles states ". . . the national profile is still consistent and often _ 
conlrasts sharply with that of other nations, according to substantial psychological test 
results and public opinion data." The sccumulating evidence is unmistakable: Over a 
span of at least 200 years there has been a marked, indeed a remarkable degree of 
cqntlnuity In the American national character. But the evidence tells us too that certain 
prominent changes are also occurring. .." (p. 26). 

One of the continuities is the continued intensely held belief In the special qualities of 
the American system--a special brand of patriotism. With 12 percent prefernng to live in 
some other country (lower than for nine European countries) the large majority were 
committed to this country even In the light of a sharp erosion of confidence in basic 
institutions, including government. 

Americans still believe in the power of an individual to shape his or her future. This 
belief is held not only by professionals and business people but regularly by two-thirds 
to' three-fourths of American blue collar workers. The emphasis on self-reliance and 
independence continues. 
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The majority of Americans believe that most people can be trusted; that we can 
transform the physical and social world and even human nature over to our own 
satisfaction. The majority have optimism in their personal economic futures; an 
openness to new expert^n^jnd innovation extending beyond the technical and 
mechanical to the social and iensate; a continuing propensity to assert their rights of 
personal autonomy over public control, Inkeles considers the evidence and concludes 
that also continuing are: "a sense of intrinsic worth and a conviction that one is equal to 
all others before the law; individualism,\ restless energy; pragmatism; brashness or 
boastfulness; this wordllness; a preference for the concrete; and a certain discomfort in 
coping with aesthetic and emotional expression" (p. 29). 

Changes according to Inkeles include: "a substantial and steady increase in tolerance of 
religious, sexual, and raciaf differences; Increase in inner-direction versus other 
direction; a shift from the number of people who considered the most important 
attribute of a job to be Its Intrinsic Importance or its promise of advancement to high 
Income and shorter hours as being more attractive. (Evidence of valuing the ability to 
purchase symbols of success, achieve material comfort and have greater freedom, 
through more discretionary time?) Seniority and experience were related as the basis for 
getting ahead in their line of work by more persons than those mentioning hard work 
and persistence. The consumption ethic seems to have replaced the Protestant ethic of 
saving and investment, as evidenced by the use of credit. Although he notes what may 
be some strain and conflict through some seeming inconsistencies, he believes that the 
changes do not undermine the foundations of the system (p. 32). 



VocationaUTaehnlcal Education as a Means of Achieving Goals 
Conionaht with Soeial and Individual Values 

Vocational iJucators, LagiilatorSj Employers 

Education, generallyi and vocational-technical education, specifically have been and 
continue to be valued means of achieving goals directly related to social values which 
take form in individual valued. Its continued support Is linked to its credibility as an 
effficient and practical means to achieving thos# social and individual ends. That 
eredlbillty is not exclusively based on hard data from evaluative research, but, presently, 
of equal or greater significance is, the belief of a sufficient number of vocationah- . 
teohnlcat educators, employers, parents, and students that it contributes significantly to h 
social and individual goals. Furthermore^ the valued social and Individual goals and the 
goals to which vooationahtechnical education is a means have remained fairly stable 
over the past fifty years, in the most general sense. Since 1963 however, 
vocational-technical education has been chargjsd quite explicitly to address itself to 
extending Its service to, more categories of persons viewed as having the right to and 
need for vocational education to further the social values of equality of opportunity 
through .accessibility. 

Among the acknowLedged leaders of vocational educators and leaders of the 
constituencies served are Charles Prosser, Terrel Bell, James Rhodes, elected official 
(businessman and legislator), John W. Thiele, and Roman Puclnski of the National 
Advisory Council on Vocational Education, and William Pierce, former Deputy 
Commissioner for Occupational and Adult Education. 
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In their work Vocational Education In a Democracy, Prosser and Allen attribute the 
development of vocational education to the valuing of efficiency applied to another 
value I e training and acquisition of occupational "intelligence" which exceeded that 
present in the old "picl< up method" or haphazard vocational training. Furthermore, 
education generally is viewed as the means for securing stability and progress in a 
democracy with vocational education a special form in its relation to the development 
of the material and human assets of,a people (p. 19), (Mastery over the external world.) 

Vocational education conserves natural resources and human resources (efficiency and 
practicality.) The first by promoting, disseminating and transmitting skill, knowledge and 
the results of invention and by conserving human effort. The second it achieves by 
promoting morale and intelligence by workers (p. 19). Indeed Prosser and Allen assert 
that no form of vocational education worthy of the name would confine itself to manual 
skill and general technical knowledge. Vocational education must also give that "special 
thinking Intelligence, '1' which functions in the given occupation" (p. 44). "This kind or 
use of man's job Intelligence is shown whenever a master of any occupation brings to 
bear all his knowledge to think his way through sorne difficulty that must be overcome." 

Merely organizing occupational experiences for training as a substitute for the old 
pick up method is some improvement. But it does not get us very far unless both 
processes and functioning facts are so taught that they give understanding to the 
worker and tiabits of resourceful thinking with these facts in the processes, situations, 
and opportunities of his employment. Only in this way can the native ability of any 
people be utilized to the full in the economic field" (pp. 44-45), 

James A Rhodes Governor of Ohio, cites as social ills (unachieved social values) 
unemployment, welfare, and lack of skills. He urges support for occupational education 
as a means to address these social ills (Rhodes 1969, pp. 13, 16). With a different 
emphasia from Prosser, though not inherently contradictory, he states that vocational 
education gives definite purpose and meaning to education by relating It to 
occupational goals. "It provides the technical knowledge and work skills necessary for 
employment, but It is more inclusive than training for job skills. It develops abilities, 
attitudes, work habits, ana appreciations which contribute to a satisfying and productive 
life" (p. 44). 

' "Vocational education contributes to the general education needs of youth, such as 
clti^zenship, respect tor others, and acceptance of responsibilities: but," says Rhodes, "it 
makes its unique coritrlbution to the field of work" (p. 44). 

Bell in 1975. in testimony before the Committee on Education and Labor, acknowledged 
the iegitimacy of assessing vocational education, in part, by the match between 
vocational education programs and employment opportunities. He maintained, however, 
•that it would be an error to judge the success of all vocational education by thn critena 
alone "We feel" he stated, "that vocational education programs should be perceived as 
an integral part'of the educational system of the country ... and as such is responsible 
to assist in increasing and improving basic cognitive skills, heightening career 
awareness. Improving the understanding of a variety of work environments and in many 
instances actually motivating students to remain in school at the secondary or 
postsecondary level as well as providing specific occupational skills" (pp. 308-309). He 
added that the two criteria of measuring success in vocational education programs by 
ratio of program completions to enrollments and the employment rate of graduates lose 
sight of these other aspects of vocational education. 
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Wh;at are the commonalities and differences between these expectations of leaders in 
vocational education and representatives of the policy and one consumer of vocational 
education namely the employer? One noted authority on business and management 
provides one source of information. Drupker (ManQgement, 1974, p, 267) brings together 
social and individual values when he describes the prerequisites for responsible workers 
and parenthetically states that the fundamental reality for every worker is the eight 
hours or so he/she spends on the job through which the great majority has access to 
achievement, to fulfillment and to community. In order to achieve, the worker must be 
able to take responsibility for the job. To do this requires: (I) productive work; (2) 
feedback information; and (3) continuous learning (p. 267). Productive work is based on 
knowledge, analysis, and skill. Feedback information about the worker's performance as 
against standards is the major reinforcer and tool of the worker for measuring and 
directing herself/himself. Relative to continuous learning. Drucker notes that continuous 
learning does not replace training. "It: has different alms and satisfies different needs. 
Above all, it satisfies the need of the employee to contribute what he himself has 
learned to the improvement of his own performance, to the improvement of his fellow 
worker^s performance, and to a better, more effective but also more rational way of 
working.'' Continuous learning comes to grips with the two basic problems of resistance 
of workers to innovation, and the danger that workers will become obsolete. It is as 
appropriate for clerical work as for manual work and knowledge work. The work group 
has to be seen and has to see itself as a learning group (p. 270). 

A survey of employers in New Jersey was reported in 1976 (Task Force on Competency 
Indicators and Standards, Rutgers Research Team, October 1976) as a part of an effort 
to identify minimum basic skill requirements for employment for high school graduates. 
Small, medium, and large firms were surveyed to explore the area of employment 
practices to determine certain basic parameters. It was found that 'Yequirements for 
employment were most typically job-related and are becoming more so as a result of 
recent court rulings. Very few general academic requirements were found although ^ 
some firms indicated they preferred high school graduates particularly for clerical 
positions. For some skilled occupations trade school training and/or apprenticeship 
training is required. High school graduation was often seen as a measure of 
dependability and 'stlck-tO'it-iveness' rather than as a guarantee of basic skill mastery'* 

Communications and arithmetic were commonly stated requirements for clerical 
positions and were considered important for a wide range of jobs as well. Firms using 
tests are more likely to be large and to have specified requirements for each job. Such 
tests are practical and job-related as opposed to general academic tests employed in the 
past since courts have ruled that tests must have demonstrated validity for the job. 

In sum, commonalities do exist among these producers and consumers of vocational 
education, however, differences on some points of significance appear— specificallv 
difference in the narrowness versus the breadth of program objectives; in one sense 
narrow training versus education for occupational activity; the explicit attention to 
educating in processes that have more enduring value as well as the immediately 
relevant job specific skills. Taking into account the customary gap between ideafe and 
practice, the narrowly conceived outcomes illustrated by the QAO study, the enri^hasis 
in the legislative mandates for preparation for employment, and the more frequently 
used outcome measures of job placement, length of time to gain employment, ^nd 
employer satisfaction, one can speoulate that the narrowView Is in all likelihood more 
pervasive, more prevalent, and more predominate in program implementation as well as 
evaluation. ' ' 



10 



The narrow expectations seem based most c\cs&\) on the values of efficiency anH 
practicality in the short term. Whereas the do alopment of '^occupational intellh 
Prosser and Allen*s term attend to achieving values of efficiency and practicality 
the long term, individual freedom, intrinsic valuing of individual personality, an 
facilitate achievement and success throuah^work. 

Parents and Students: Consumers With Values 

Rokeach (1973) defines values as the cognitive representations ana transformations of 
Individual needs and societal and institutional demands. He thus clarifies and adds a 
dimension to Maslow's hierarchy of needs as motivating forces as he proceeds to 
differentiate terminal and instrumental values. Terminal values are end states, while 
instrumental values are modes of behavior some of which are moral values, others of 
which are competence values. Value systems refer to the ranking of terminal values 
along a continuum of importance; instrumental values are viewed as being ranked also. 
Rokeach found that adult men and women ranked the following terminal values highest. 



Rank 



Men Women 
665 744 



.1 World at Peace World at Peace 

2 Family Security Family Security 

3 Freedom Freedom 

4 Comfortable Life Salvation 

5 Happiness Happiness 

6 Self-respect Self-respect 

Instrumental values ranked as the top six were; 

Rank Men Women 
665 '744 

1 Honest Honest 

2 Ambitious Forgiving 

3 Responsible Responsible 

4 Broadminded Ambitious 

5 Courageous Broadminded 

6 Forgiving Courageous 

As Williams noted, and Rokeach implemented, detailed descriptions of values must take 
into account various social categories such as income, education, race and age. An 
analysis of values compared by Income and then by education found that persons of 
low income and limited education held much the same values some of which differed 
from those with high Income and higher levels of education. A world at peace, family 
security, and freedom ranked high rather consistently across all categories, A 
comfortable life, self-respect, and friendship tended to be ranked higher by those with 
lower Incomes and less than high school education than were ranked by those with 
higher Incomes and who were high school graduates and above. Instrumental values 
ranked high by those with less than high school education were: honest, clean, 



11 



1 



ambitious, forgiving, helpful, and courageous responsible was ranked ^ ^ 

those havmg completed some high school Relative to race, when ) 

and education were held constant, the one JMierence was the higher rann 

accorded equality, second for blacks and twehu i for whites (Rokeach, pp. 57=72 

For these data to have any import for the concerns of this paper, it is necessary to make 
several assumptions. First we must assume that Rokeach's data have some limited 
generallzabillty. Second, we must assume that men and women look to occupations as 
one means for achieving these values, with the corollary that vocational education 
programs interpreted in terms of these values would be more highly valued as a means. 

If these assumptions are at least reasonably tenable, then the findings have some utility 
for conceptualizing outcome measures which relate to the efficacy of vocational 
education in achieving goals derived from the^e values. 

In an effort to gain insight into the valuss of secondary age students, the above data on 
adult values have utility only if we can accept the premise that these adults as parents, 
will desire for their children some of these same values. If so, then interpretations of 
program ctiaracteristics, goals, and evidence of outcomes could be developed with close 
attention to'^the conditions and behaviors which are associated with attainrhent of these 
values. The recent emphasis on educating in life skills for productive participation in 
society, as a high school graduation requirement lends some support to such a premise. 

As for secondary age youth, themselves, Rokeach provides some data, Inconclusive and 
limited how^^er, regarding the salience of particular values for different ages at different 
developmental stages. Terminal values ihat generaMy are ranked in the top eight by 
eleven, thirteen^ fifteen and seventeen year olds are; h wo-^fd at peace; freedom, family 
security (lowest for seventeen year old^); friendship; euuiilKy; happiness; and a 
comfoptflble life. Relative to instrumental values* hone&i* iDving; forgiving; rank high for 
eleven* thirteen, and fifteen year olds. Some significant dep&ilures between eleven year 
olds, thJrteenj fifteen and seventeen year olds occur with increased value placed by 
these order 'groups on being ambitious, and respcnsble ana then for the fifteen and 
seventeeb year olds, being capable and Independent 

Maslow^S hidrarchy of sets of nreds may be usefuUn analyzing some of the stronger 
needs In relatlon.to specific age groups of students, 

These sets are, in oM^t nf lower ard higher needs, physiological; safety; belongingness 
and love; esteem; and sc-h actualization. Examined In relation to Rokeach's findings, we 
find sontf congruence bh^vjmn terminal snd instrumental values and the^irst four sets 
of needs. ^^Ith the fifteas : a? 'd ssventeiin year olds moving nearer to that age, whern a 
greater degree of economic self-reliarMQ i^ socially approved, terminal and instrumental 
values corns Into play tn^ii c:.^ntribute ij ^^^w^ng one's own physiological needs and 
safety, as well as enhancing 3^ :^Bm. 

Any work on values empr ':iiiRS :^ ^onipse^ ^nv^rror* cr^ between value complexes as 
motivating force In choosing aincn;i .Mtarria^^ p ngur^H^ of action, objects, or goals. 
Hence to treat terminal and instrumvniiil v :h:^n 4 .^^ngular even for purposes of 
analysis, introduces ihe iiazard of ciir,c;^f:^fcus oversimplification. For in the real world of 
action, conflict between value s^tii can tiFiid does occur, and various combinations of 
choices can lead to a pVn^ntial optimising of satisfactions related to several values. As a 
useful illustration, le^ us oonsider the generally tow status of vocational education. A 




status conceded and deplore. ' Mjch statements, frequently quoted, that we must 
eliminate the belief, too frequently held, "that vocational education is for other people's 
children!" A proposition which is acf;orded the status of almost, if not quite, being a 
sociological principle is that the status of a service group, e.g., educators, social 
workers, physicians, reiiects to some degree the status of the group served. Hence, 
teichers prefer to teach children from affluent homes, physicians are more numerous in 
affluent communities; pnd social workers, serving the poor, and disenfranchised have 
relatively Ipw status arpong professionals and salaries reflect this. Vocational education 
prepares individuals for occupations! that tend not to be lauded in the highly visible 
world of the mass media. Indeed with the differentiation of work and Its being closeted 
in plants, factories, and offices, general knowledge about these occupations, as well as 
some higher status bnes^ is relatively sparse except for those areas in which friends and 
relatives are employed. The fact\that secondary youth have been well socialized to this 
status dimension was demonstrated in studies of adolescent choices for work. Several 
researchers concluded that based on the distribution of workers in the work force,- the 
education and ability levels required, the cnoices of large numbers pf adolescents had 
to be labeled, in the aggregate, asj unrealistic since professional work was cited by large 
numbers. For the field, at large, thb relative status may, in part, contribute to the lower 
level of support frorr federalsources when compared with support for higher education 
relative to numbers served, although It Is vocational preparation for what are generally , 
higher status occuaptions. 

If we review the major value orientations delineated by Williams, such as achievement- 
success: activity-work, material comfort, and freedom through personal autonomy, I 
believe that the degree of attainment of these values in generally accepted social terms 
are optimized In some of the higher prestige occupations. Pehaps even more important 
is that most people implicitly accept the premise that the best optimizing occurs though 
involvement in higher status occupations. To the extent that these are valid 
observations. It follows that in choosing vocational education as the means for attaining 
some of these social values manifest as personal needs and values, many individuals 
have moved from the most preferred choices to the less preferred choices. An important 
qualifier to all of this, however, Is the proposition that membership in socioeconomic 
■groups tends to place a ceiling upon aspirations, and has been viewed as a social 
phenomenon that helps to reduce extreme feelings of frustration and deprivation. Thus, 
parents and family friends continue to be role models and prime influences in choice Of 
and employment in occupations. Even given this qualification, there Is evidence that a 
number of parents derive less pride from their children being enrolled in vocational 
aducation than in their continuing in a general or college preparatory curriculum with 
little or no prospect of further education in the offing. Given these conditions, there would 
appear to oe much merit In Interpreting the value and outcomes of vocatjonal education, as 
rrieans to these valued ends, quite explicitly for parents and students. There is, I 
suggest, too little mention of the likelihood that vocational education for some is 
preparation for vvork which will be transitional and In turn a means to other statuses. 
Such an approach may engender some risks given the QAO and support in some 
quartera for the position that vocational education can be Justified almost exclusively by 
job placement-and that in occupations for which trained, or\in a related occupation. 
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Outcomt Measures in ivaluating Vocationai Education 

Evaluation and evaluative research need to be differentiated for fruitful consideratio^H 
^ within the parameters of this paper. Suchman (1967) proposes the "distinction bet\A/^%n 
evaluation as the general process of judging the worthwhileness of some activity 
regardless of the method employed and evaluative research as the specific use of ti^# 
scientific method for the purpose of making an evaluation" (p, 31). He continues th^'t 
range of variation can be indicated by **defining evaluation as the determination 
(whether based on opinions, records, subjective, or objective data) of the results 
(whether desirable or undesirable; transient or permanent; immediate or delayed) 
attained by some activity , , , Designed to accomplish some valued goal or objoctivl 
(whether ultimate, intermediate or immediate, effort or performance, long or short 
range)" (p. 32). Suchman points out that the evaluation process moves from value 
formation, to goal setting, to go/il measuring, identifying goal activity, to putting ggAI 
activity Into operation, to assessing the effect of this goal operation, and back to vg'u^ 
formation. 

Specifically, what values are addressed in the outcome measures recommended for UBe 
in program evaluations and system evaluation, derived^ by aggregating outcomes frc^tii 
.program evaluation? The source for these recommended outcome measures is ''A 
System for Statewide Evaluation of Vocationai Education," The Center for Vocationl*! 
and Technical Education. In broad categories, these outcome measures include: (T) 
descriptions of characteristics of individuals served by the program and, in aggrega%, 
the system; (2) the successful completion or early departure from the system; (3) 
acceptances-exclusion rateo; (4) employment and earnings history and current statt/^; 
(5) aspirations for further education; and (6) satisfaction with program. Among the 
major value orientations evident in the choice of these measures are: attainment of *V^rk 
versus unemployment; earnings are one indication of level of material comfort, 
achievement and success; satisfaction levels and interest in further education 
acknowledge the intrinsic worth and a degree of personal freedom through autonon^y of 
choice and furt^ " development; admission* exclusion, characteristics of persons se^w^d, 
length of involv . lent, and conditions of departure address the value of equality of 
opportunity through evidences of accessibility. Attention to the means value of 
efficiency and praGticallty underly the data collected regarding program length, us© W 
advisory councils for evaluation* current manpower data for program planning, facjilH^s. 
and inclusion of cooperative work experience or ^^mulated work experience. 

Statewide system evaluations are supplemented by program and local district 
evaluations. Some such efforts have Included measures of attitudes towards work, 
work-related behaviors, self-concept, source of interest fn work, changes In self- 
evaluation In relatloato work (Nelson and Jacoby 1967; Kievit 1973), These outcome . 
measures addreb^ more specifically the major value orientation of freedom through 
personal independence, self-confidence, and autonomy, and the intrinsic worth of 
individuals as warranting development in the broader aspect of Prossfer's "occupatio'l^l 
intelligence/* These are at the most elementary level, however, and limited rather th#ri 
coniprehensiye. Although the number of sources of outcome measures used has bi#ri 
limited, this primarily grows out of my immersion in evaluation In vocationai educatl^h 
which leads me to believe that these outcome measures accurately illustrate the ranft^ 
of measures most frequently used. Outcome measures which address individual yBlJm 
related to freedom, achievement and success defined in individual terms relative to ^ 
aspiration levels, and Individual personality such as of intrinsic worth ar^ employeq i>M 
systematic .evaluative research less frequently and in more limited evaluative studies^ 
These are more likely to be addressed in evaluation which includes heavy reliance ub^n 
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informal observation, subjective assessments of spontaneous expressions of students, 
parents, and employers. Evaluation at this level is more likely to occur without explicit 
awareness of the full range of values which could potentially be furthered in attainment, 
and with little attention to the nuances and shadings in communication with members of 
various constituencies which point to the values rnost salient to that person at 'the time. 

The value of the initial distinction between evaluation and evaluative research resides in 
facilitating greater clarity regarding the purpose of evaluation in a specific situation. and 
thi Intended consumer of the report as this has implications for choice of outcome 
measures and data to be collected. Administrators, program planners and evaluators, 
and teachers should be a team of producers and consumers of evaluative research. This 
team should be used as a basis for fine tuning the organization of educational 
exf)eriences to provide the optimal outcomes in terms of manual skill, technical 
knowledge, and occupational intelligence. In addition, administrators, counselors, and 
teachers are strategically placed for explicitly linking program objectives and outcomes 
with individual values of students and thetr parents. Administrators at the local level and 
particularly those at the state level need to be attentive to interpreting the outcomes of 
system evaluative research in terms of valued means which characterize vocational 
education, and the valued ends to which vocational education contributes. 

The choice of outcome measures should be reexamined, and expanded to more 
specifically seek information regarding efficiency and practicality not only in the short 
term but also over the long term. The contribution which vocational education has made 
to the individual's sense of attaining, to some measure, the values of family security, a 
comfortable life, a sense of expanded freedom through expanded options among which 
to choose, with a sense of confidence, self- respect, equality, and social recognition 
should be documented. The issue is not whether these values might not have been 
attained by other means-they probably would have to some degree but rather to 
demonstrate that vocational education is one means among others, equally effectiv m 
most cases and more effective than other means in "■some cases. This type of 
documentation would be pr., ticulnriy n.h as a basis for information to disseminate to 
parents and potential students, as well as legislators concerned about the opmions and 
views of their constituencies. 

Interpretation of outcomes to thin last group might be more effective if couched in terms 
of equality of opportunity through differentiated and expanded options for many: the 
sense of success by virtue of having the capacity to purchase tnore of the symbols of 
success, through higher earnings, even if the work is mid-ranged to low in prestige. 

Documentation of this nature would seem to call for tracking of vocational education 
students through time, at various regular intervals. The case study method with 
Interviews would afford the greater likelihood of gaining insight in the contribution of 
vocational education to value attainment. The diversity of the life styles surfacmg, the 
interplay between work and education, job changes with impact on value attamment, 
would be informative and add much to our understanding of the impact of individual 
programs and state systems. 
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Legislators and Board Members: ^ 
Audience for ivaluation Research and Evaluation 

Legislators and board members are concerned about accountability to their various 
constituencies. One criterion for accountability is her/his effectiveness in promoting and 
sustaining federal* state, or local policies and programs which are perceived to be 
beneficiaL Benefits are related to the achievement of goals linked to values. In view of 
the preceeding examination of values, it seems reasonably evident that values most 
salient for these groups include an employable rather than unemployable constituency, 
constituency material comfort vs. material impoverishment, independence of individuals 
vs. dependency, and equality of opportunity, and equal access to programs. Credible 
documentation that vocational education is contributing, efficiently and practically, to 
attaining these ends for an important segment of a legislator's constituency will provide 
a more persuasive rationale for support. Evaluative research and evaluation should be 
the basis for providing this documentation. 

Evaluative research should include the placement, Job satisfaction, admission and 
completion data elements as well as the others cited earlier. It should be supplemented, 
however by some systematically obtained case materials which provide "close-ups" of 
what the data mean in individual and human terms. Data obtained from case studies should 
be targeted specifically on the values of Individuals enhanced by vocational education. The 
results should be couched in "value" terms when reported. 

Report format might be modeled (with some adaptation) after the annual financial 
reports of a large corporation i he better ones of these show the statistical data - 
succinctly, and elaborate on the meaning of these statistics for the central concerns of 
the firm. The narrative could include the **Glose-up'' of individuals and groups served 
with some general statements indicating the typicality of the "close-up** for the total 
population served. Goals only partially achieved, but the subject of Imaginative planning 
and continued pursuit, should be Included and clearly presented as evidence of shared , 
concern for efficiency and practicality as a means to the end values. 



imployers: Audience for Evaluations 

Employers value productive workers as essential to increased efficiency. Hence, 
evaluations should attend to those program outcomes which are linked to student 
behaviors essential for rapid integration IntQ the Job setting with full productivity within 
the shortest time period feasible: habits of vvork, such as knowledge of safety measures, 
dependability that reduces loss of time, a fairly precise indication of level of skill , 
performance to be expected from specific vocational programs, and the adaptability and 
capacity to learn Is essential for responsible workers. The values of employers which 
vocational education contributes to are comparatively narrow. 

r Modes of reporting might well be in the form of an Invehtory of these behavioral 
^(Outcomes from ^ ocational education programs, The proportions of graduates who 
achieved minimum levels, clearly defined, in the various areas could be reported in such 
a format. 



Administrators and Teachers- 
Audiences for Evaluative Research 



Evaluative research should be designed to linK assessments of outcomes to the 

attainment of individual values of students, both present and emerging. Close attention 
needs to be given to the controllable variables which may be significant points of 
intervention for teachers in structuring curriculunn, classroom managsment, and 
instructional strategies to increase the efficiency of learning for many students, 
increasing the success rate for those admitted, and enhancing attainment of the 
personal values of students for (a) feeling capable, (b) achieving a sense of esteem from 
self and others (c) sharing friendship, and (d) increasing individual freedom through 
decreased dependency and Increased capabiiity for independence in a number of 
spheres through the attalnmant of skill, problem solving abilities, and earning potential. 
Setting goals and monitoring progress or lacK of progress toward these goals are 
essential to evaluation of ongoing program implementation designed from evaluative 
research Routlnlzed feedback should be an important part of the process. The feedback 
should include summarizing q!i|.rterly reports and an annual report. Some quantmable ■ 
data of use would include attendance rates, dropouts, meapures of skill performance in 
vocational area, anecdotal observations regarding problem-solving skills and those 
behaviors indicative of goal attainment relative to the values cited above. 



Students and Parents 



The values of individuals to which vocational education is one means include family 
security freedom through economic independency, self-respect, and material comfort. 
Work and preparation for work are processes through which individuals develop and 
exercise friendship, responsibility, capacity and ambition. Evaluation which is couc_hed 
In "close-ups " vignettes, and anecdotal types of communication can effectively link 
outcomes of vocational education to individual values. A series of well-designed, 
colorful posters, each focusing on a single value and multiple program processes and ^ 
outcomes whic'^arrTBlated is one reporting format. Spot public service announcements 
on radio and brief film clips on television designed on the same rationale could be more 
widely disseminated. News releases in local daily and weekly newspapers leading ott 
with a "close-up" for human interest on one or two values and concluding with 
statistical data regarding the typicality of the achievement could also be effective. The 
focus and reporting made could be similar for parents and students with slightly 
different points of emphasis. Young students in secondary programs are less concQrned 
with family security but are concerned with acquiring freedom through indepenuence, 
being capable and developing friendships. Data obtained through case study and 
interview are best suited for dissemination to these groups, In conclusion, as vocational 
educators concerned with linking programs and program outcomes more closely to 
related values of our various audiences, we should direct attention to ways and means 
of aseertaining those values of persons and groups of immediate concern. This paper 
nrovldes a general framework, a spring board, so to-speak, from which to be better 
attuned to appropriate starting points in seeking information on values. It is the ^ 
broad-brush stroke and not the fin !y detailed picture needed for most effectively inkmg 
particular programs in vocational education to values-social or individual. To seek such 
linkages is a worthwhile direction. For in the process, individual purposes and legislative 
Intent may be more clearly articulated, more subject to reexamin. tion and positive 
modification. Vocational education may become more effective as a means to attaining 
social and personal values. 
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OUTCOME MEASURES RELATED TO INDIVIDUAL AND SOCIAL VALUES 
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EVALUATING VOCATIONAL f DUCATION: 
A CONGRESSIONAL PERSPECTIVi* 

^ John Jennings 

Tonight. I would like to talk about two things: (1) the content in which the new 
evaluation requirennants for vocational education came about in the Vocational 
Education Amendments of 1976, and (2) the nature of the requirements themselves, 

I know you are familiar with the history of vocational education so I will not review its 
development all the way back to the congressional enactment of the Smith-Hughes Act 
which initiated vocational education as we know it today. I will skip the first years and 
start with 1963 when a landmark vocational act was passed. The 1963 Act had the effect 
of broadening federal support for vocational education, and focusing more attention on 
it. The 1968 Amendments further broadened that Act and brought about some updating 
in programs through authorizing re&oarch funds and exemplary programs. These 
amendmefrts-also focused attention on special populations such as the disadvantaged, 
the handicapped, and on postsecondary students, whose needs Congress felt at that 
point in time were not being adequately addressed. Then, more recently, the 1976 
vocational education legislation was passed which is why you are here tonight. 



The iducatlon Amendments of 1976 



The 1976 Amendments build upon what happened in the past but they must also be 
viewed within their own context. During 1975 and 1976, Congress held very extensive 
hearings and found that the data showed vocational education to be very successfuK at 
least, In quantitative terms. There seemed to be many more dollars being spent at the 
state and local levels for vocational education than ever before. There seemed to be 
many more facilities, more students enrolled, and some empirical evidence that there 
was success with those students in the program. However, the legislators in reviewmg 
the progress of the Act were confronted with several problems. 

As I describe these problems, I would like to emphasize that I am in effect summarizing 
the House and Senate committee reports on this legislation since committee reports are 
frequently the best, and sometimes the only, documents which give a thorough 
explanation of the congressional intent surrounding legislation. 



* A transcription of a presentation by John Jennings, 
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Problems Addressed by the 1976 Legislation 

Isolation 

In 1976, these committee reports stated a general aonclusiuii that vocational education 
prQgrams as administered at the state leve] in many states seemed to have too much of 
a tendency to be isolated unto themselves. In state departments of education, vocational 
educators sometimes were isolated from other parts of the state department. Within the 
states ihemselves vocational educators were too isolated from postsecondary educators: 
and also, vocational educators were too isolated from the people operating ComDrehensive 
Employment Training Act programs and other types of training programs. And, of course, 
the same w^as true in reverse: these other people who administered programs yery 
similar to vocaiional education frequently isolated themselves from vocational 
educators and others administering similar training programs. 



Use of Federal Funds 

An additional problem seemed to be that there was difficulty in finding hovv the fund!i 
were being used, especially the federal money. This was true for a variety of reasons, In 
many states, federal dollars were mingled with state and local dollars so that you really 
couldn't tell what the foderal dollars were bringing about or even paying for Although 
the state plans were supposed to show the expenditure of federal money, they really 
didn't show much because they were merely compliance documents completed to fulfill 
the exact letter of the law. There rJso seemed to have been a problem with the attitude 
of the federal adminstrators in that they operated the program as a revenue sharing 
program without going out to the states to give them assistance or trying to correct 
misuses of federal money that occurred. It appeared that generally federal money was 
being used properly, but possibly it wasn't being used properly in some isolated 
Instances; and no one knew tor sure how many of those instances there v^ere. 



Lack of Good Dato 

Another problem seemed to be a lack of good data to show exactly what was 
happening, not only with federal money, but with state and local money as well. In one 
state, they counted anybody who was In career educatjon as a vocational student, in 
other states they did not. In some states they counted students as vocational students if 
they were In one course for orie hour a week, and in other states they did not. In some 
states you had to be a fulhtime student In a course for many hours, and in other states 
you could be a part-time student. Differences also existed across states as to the types 
of credentialing used. There seemed to be no uniformity In the data, even .sometimes 
within a given state. So, if you were to take the testimony of people who said that things 
were going wall, you couldn't really find out for sure because there was not the solid 
data base to back up those statements. 
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Lack of Follow-up 



The last of what these committee reports called pro^Jei i had to do with the inability to 
look at the quality of the programs. There were sople witnisses who were able to show 
through follow-up studies that their graduates had done well. But this type of foUaw-up 
seemed to be very sporadic. You had somebody from a city saying: "We have this 
record that kids did well/* But you really did not have any uniformity in the following up 
of tnose individuals over several years or data from other school districts in terms of 
whether that was true for different types of programs or even the same type of program 
over time. 



The Congressional ConteKt 



What Congress tried to do was to concentrate on these problems and bring about some 
solutions. At this point, I would like to repeat what the reports stated in describing these 
problems, namely that this description of problems was not meant to leave a negative 
impression of vocational education; rather, the Congress v^as trying to discover what the 
impediments were to a better vocational education system and was trying to help 
remove those impediments. It v^as presumptuous in a way for Congress to try to address 
these things because the federal commit.ment to vocational education, even though it 
has grown in dollar terms, has in no way been able to keep up with the expansion of 
state and local funding. Though state and local money grew by 100 percent between 
1971 and 1976, the federal appropriations increased by only 37 percent. Obviously, the 
federal government was by far a Junior partner. 

Nonetheless, Congress felt for several reasons that it should be bold in trying to say that 
certain things should be put into process. Congress was the first in trying to encourage 
vocational education in a national sense through enactment of the Smith-Hughes Act in 
1917, And since then, Congress in this field peculiarly has been the body which has 
given some direction to vocational education, frequently out of proportion to the federal 
money which was being made available. Sometimes vocational educators look to 
Congress and are willing to accept solutions even though Congress isn't providing the 
bulk of the money. 

Further, the friends of vocational education In Congress felt that the ball game was 
being lost because within that same bracket of time when Congress was minimally 
increasing vocational education approprJations, barely keeping up with inflation, the 
appropriations for the Comprehensive Employment and Training Act (CETA) were 
growing by leaps and bounds. That program has gradually changed from a job 
retraining program as it was first constituted in 1962 to a public service employment 
program today with what seems to be a very low priority on Job training. Yet, billions of 
dollars were going to the CETA program; and some Cwngress; arsons felt that 
vocational education had to show a better record in order to be able to receive some of 
that funding. The congressional friends of vocational education felt that a better case 
could be made for the training provided by vocational education because that training 
produced a long-term productive solution instead of a short-term reactive solution. 
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In the Amendments of 1976, Congress tried to help vocational education make this case. 
First, more administrative flexibility was given to the states through "block>granting" 
most of the funds. But, then, second, the legislation concentrated on making changes in 
the way decisions were made by the states, In other words, the legislation concentrated 
on the decision-making processes at the state level. It seems that in vocational education, 
and, again, rather peculiarly, the state departments of education have quite a say in areas 
in witich they doi't always have a say in other types of programs. Consequently. 
Congress tried to impact on the decision-making at that level in four areas. 



New Amendments 



Planning Requirements 



One area of change involved the planning requirements. They w are to be very ■ pecific. 
In this Act, Congress was extraordinary in writing out exactly how the planning process 
had to occur at the state level. The law specifies the exact agencies within each state 
which must participate in the planning process. The law also says that there has to be 
so many meetings a year and describes what has to happen in each one of those 
meetings. That is extraordinary for a federal law. I think Congress felt a need to open 
up the decision-making process so that all elements of job training could get involved in 
the way in which federal funds were being spent. 

Accountability 

Congress also required that the state plan, which was the document to be used for this 
decision-making, had to be very precise and had to say exactly where the money was 
going, and most Importantly why the money was going for certain things. Now I will 
dare you to go back and look at state plans which were complied in vocational 
education for many years to find these things, and you will not find many plans setting 
out the exact reasons why certain things were done. It seems a lot of things that were in 
the law were simply regurgitated in the state plans; and that document, in fact, was r ot 
a planning document that presented reasons why things were done or not done, 

Improvtment of Data 

Another thing Congress tried to do was to include provisions in the legislation to 
enhance the chances for better data. In order to achieve better data, federal data 
gathering was transferred from the U,S, Office of Education to the National Center for 
Educational Statistics (NCES). And, the legislation also laid out the specific elements of 
data which had to be collected by the states and from the states. Congress also 
mandated the creation of the National Occupational Information Coordinating 
Committee and companion state committees to help the states and NCES with this task. 
These committees were also to work on achieving more uniformity in data gathering 
among all training programs and to work on achieving the use of better occupational 
demand data in those programs. 
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Evaluation Requirements 

You are aware, of course, of the state evaluation requirements which are the last area 
with which the new legislation deals. Congress is saying that within the five years of its 
state plan each state has to evaluate all its programs. Evaluations of specific elements of 
programs offering entry-level training are requireci, i.e., as to what happens to the 
completers of those programs in terms of whether they achieve employment in Job 
related fields and what their employers' opinions are of their abilities. 

I want to discuss why Congress chose those criteria and why it did not choose other 
criteria. I think there was a feeling that a program which claimed that it was training 
people to send them out into the job market should minimally show whether the people 
who were trained in that particular skill did or did not obtain employment in that field, 
and secondly, what the employer thought of their capabilities. 

There are many other things which could be taken into consideration because you 
cannot take human life and reduce everything to those two simple facts in order to 
judge "success" or "failure." There are always so many things in life, so many variables, 
that you cannot simplify and understand them all. But, it would seem that if there were 
any basic information that would serve as a beginning point to evaluate this type of 
program, namely a program claiming to offer entry level job training, it would be 
whether people became employed and what their employers thought of their training. 
Now perhaps if there is a low placemen , racord; there are good reasons for that and 
those can be explained. Maybe there are particular characteristics of the program that 
make that type of criteria inappropriate and that can be explained also. Yet, it would 
seem that you have to start somewhere in looking at a program to determine its 
usefulness; and I think that is why Congress settled on starting right there. 

Parenthetically. I would like to deal with a criticism I Have heard of congressional action 
with regard to this new requirement. Some people say that if Congress wants to have 
such an evaluation made. of the uses of federal funds for vocational education, it should 
mandate the same requirement for the programs of higher education supported with 
federal funds. Although on its face this criticism has some merit, it must be remembered 
that there is a basic diffe];ence in the manner in which Congress provides funds for 
vocational education and'ifor higher education. 

Vocational education really receives general institutional assistance with certain broad 
requirements placed on its use. Higher education does not receive institutional support; 
rather, the vast bulk of federal support for higher education goes out as grants to 
students and then each student decides where he or she wants to attend school and 
brings that assistance with him or her to that college or universityl So, federal support 
for higher education has an inbuilt "quality** standard by relying on the judgments of 
millions of students as to which institutions they believe are the best for them or the 
most appropriate for them. 

Vocational students cannot make that choice since the federal funds are given to state 
vocational administrators who grant them to local vocafional schools. So, the ^'quality" 
check in this situation Is now meant to be how these schools' programs fare in the . 
newly required evaluation. 
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Implementation 



NIE Report 

Recentiy» I was given a draft copy of a report to the National Institute of Education on 
the status of the states' (mplementation of these new evaluation requirements specified 
in the 1976 Amendments. That report states the following conclusions. 

First, it seems that apprOKimately a third of the states are using their present systems for 
evaluation to comply with these requirements. Some of those states say that their 
systems comply, and others ^ay that they do not wmt to change their present systems 
very much and so they will s^ay with what they have. There seems to be a second group 
of states, another third, which has gone about trying to change their present systems in 
order to bring them more in line with the spirit of the law. And finally, there is a third of 
the states which is just starting to implement some evaluation features. Some of them 
have started pilot prQcedures and are moving on to regular. procedures next year. 

Consequently, it is apparent that the states have a way to go before they will be able to 
meet these new requirements in full. It is understandable that the states are in this 
situation because up to now there have not been those types of strict requirements. But 
the states must continue to press on with their implementation if they really care about 
fundin^^he programs which will provide the best poasible training for their citizens. 



Furthermore, the results of these evaluations are meant to mesh with the other three 
types of req^ements which were put into the Vocational Act in 1976. If the evaluation 
data obtained are good, they should be put into the planning process so people will 
know what type of programs are successful and which programs are not successfuL It 
would seem that if one secures good evaluation data, that type of data can be used at 
all different levels of decision making. In so doing better planning, greater 
accountability, improved data and good evaluations will mesh together, hopefully to 
bring about better programs in vocational education. 



Title I Evaluations 

The last thing I would like to discuss is what has happened with Title I evaluations and 
how that might have some meaning for what you are embarking upon right now. If I 
understand a recent report correctly, it seems that a strong effort to do evaluation on a 
comprehensive scale is a new thing In vocational education in most states. To do 
evaluations in a systematic manner seems to be very new for the vast majority of the 
states. Therefore, something might be learned by looking at what has happened in 
another education program where these types of evaluation requirements have been in 
effect for a much longer period of time. 

Title I of the Elementary and Secondary Education Act of 1965 is the major federal 
program, of aid to elementary and secondary education, involving approximately three 
billion dollars of federal funds for compensatory education for poor children. There have 
been legislatively required evaluations for Title I programs since the mid '60s. Title I was 
the first major social program which Included requirements that state and local 
evaluations of the programs be conducted. The amount of money spent on evaluations 
of that program has gone from one million dollars to over twenty million dollars in the 
last several years. 




26 



EKLC 



A report that wl^s just done by the Stanford Research Institute in Palo Alto on what has 
happened with the evaluation requirements in Title I is directly relevant to what you are 
discussing at this conference. The first thing this report points out. and I know it to be 
correct, is that the congressional intent behind the Title I evaluation requirements was 
that local people should look at their programs periodically and try to evaluate them to 
see what could be done betUir and, hopefully, to put those improvements into their 
programs. From the beginning, the feeling on the congressional level has been that the 
primary purpose of these requfrements is to try to bring about local program change. 

Ti*"fis local perspective for the requirements was not carried Dut by the federal 
administrators of the program, instead, the U. S. Office of Education went In the 
opposite direction in implementing that provision. Almost from the first days of the act 
the U. S. Office of Education tried to.get nationally uniform data, data which could be 
aggregated at the state level, then aggregated at the national leveK This would enable 
them to come to Congress and the administration and to tell them exactly the n jmber of 
successes or failures in the national program. The only way they could see themselves 
doing that was to more or less impose, although sometimes indirectly, the use of 
reading achievement tests. 

Consequently, what one finds in Title I, as you go all the way down the line, is that 
reading achievement tests are for the most part, used as the only criterion for success in 
that program. What that has meant is that what is being required nationally in order to 
achieve some type of national data has resulted in these evaluations having almost no 
influence on the local level in terms of changing those programs. In other words, it 
seems that the requirement that started out initially with a focus on the local level has 
been thwarted over the years in an attempt to get national data. What has been achieved 
to date is pretty well irrelevant to the people back on the local level. It may or may not 
be relevant at the national level. 

Congresspersons are very happy to hear that the reading achievement scores in Title I 
have gone up by so miany months because they can claim that is a wise expenditure of 
money and they like to go to the appropriations committees and tell them they have to 
put more money into the Title I program because the scores have gone up. And yet, if 
that is the sole use of the money, or the sole use of the evaluation data, it seems to have 
frustrated the primary purpose of the requirement. 

You shou d keep this Title I evaluation phenomenon in mind as you try to implement the 
new evaluation requirements of vocational education. If all of you work for several years 
to secure some type of data which may or may not be accurate, but which can be 
aggregated and can be sent on to Congress, then you will not have carried out the 
primary purpose of the evaluation requirements. The primary purpose is to try to give 
local people, and state administrators, an opportunity to learn how their programs are 
operating and to help them improve those programs. 

This local purpose for evaluation may be difficult to carry out because, as the report on 
Title I points out, there are several reasons why local people say that Title I evaluations 
are pretty much irrelevant to their decision making, even in addition to the nationally 
required use of achievement tests. These reasons directly impinge upon what you are 
embarking upon. These factors are: 
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1. Tht stability of the program. As with many programs, Title I programs hve on 
and on with a certain momentum to them. And if that is true in Title I, you can 
bet that it is true with vocational education, possibly to a greater degree, 

2. The timing of evaluation. It seems that most of the evaluations produced in 
Title I just come at the wrong time of the year for them to be of much use to 
people who are reviewing the local programs. Again, that is directly relevant to 
vocational education. 

3. Minimal l|nkaga between evaluators and administrators. There is very little 
connection in most school districts between the Title I evaluators and the Title 
I admininlstrators. This is especially true when outside evaluators are used in 
Title L It seems that what they produce is not of great relevance to the people 
who really make the decisions in how to shape Title I programs. 

4. Many diverse audiences. Title I has many different audiences and these 
audiences have different perceptions of what the programs are supposed to do. 
Also, they have dlfferer^t ideas of what type of information should be produced 
in evaluation. A number of audiences feel that achievement te^i scores are not 
a valid criterion upon which to solely judge the program. That problem of 
course is one vocational educators face very intimately in trying to judge what 
type of audiences are to be served, what type of criteria should be used 

5. The state of the art of evaluation. This is a developing field and currently 
involves many divergent evaluation strategies. 

6. Evaluations are perceived at the tocal level as a threat. If an evaluator reports a 
reading score on a Title I program which has not gone up to a certain degree, 
the Title I teacher or administrator has a fear of being called in on tht' carpet. 
Vocational educators face that same problem. 

7. Problems are explained away. Regardless of what problems are contained in an 
evaluation report, the Title I staff is usually able to "explain away*' why those 
things are there and be able to point out that one is not really looking at the 
right things and that there are other things to be considered, 

/ 

Those seven things are more or less things which everybody dealing with evaluations of 
programs is going to face, and the task is not easy. 



Summary 



This conference is a beginning of a significant discussion of where we should go with 
vocational education evaiuation. There are so many variables and so many differences 
of opinion. A lot of things are going to have to be discussed and discussed over a long 
period of time so that we can gradually work our way towards a valid system of 
evaluation. In my opinion, the evaluation requirements in the Vocational Education Act 
are probably the most significant requirements in that Act, because over time they 
potentially could have the greatest effect on programs. 
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A number of vocational educators say that they cannot terminate local programs tpven 
when employment demand and placement for certain progrnms is quite low. For 
example/every high schooi wants to have a beauty culture caurse and there is no way 
state or local officials can go to those people and say your students are not being 
trained in an area of need. If you want to have that course, you must pay for it out of 
your own money. People at state and local levels have difficulty doing that now partially 
because the data are just not there to show that is so. 

Since we are beginning to face a situation where new dollars for education are going to 
be hard to come by, we must have some idea of which programs should be supported 
with our funds. Potentially, these new evaluation requirements in vocational education 
can give us some facts on which to base our decisions. And, as citizens, I am sure that 
we all want the most judicious use of our tax dollars. And as citizens, we want to make 
sure the programs offered are going to give students the best type of job training 
opportunities. You have an opportunity at this conference to debate those issues and to 
give us the beginnings of some answers. 



John Jennings received a B.A. in history from Loyola University and J.D. from 
Northwestern University School of Law. Since 1967, Mr. Jennings has served in his 
current position as Counsel and Staff Director, Subcommittee on Elementary. 
Secondary, and Vocational Education of the U.S. House of Representatives Committee 
on Education and Labor. The subcommittee has jurisdiction over acts dealing with 
vocational education, higher education, and other legislation affecting elementary and 
secondary educaion. 
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OUTCOME STANDARDIZATION FOR COMPLIANCE 
OR DIRECTION: THi CRITICAL DISTINCTION 



Donald W. Drewes 



Standardized outcome moasures--rgin or salvation. Although the statement is reminis- 
cent of sensational joufnalism, it does serve to anchor the extreme points of opinion 
with respect to the standardization of outcome measures in vocational education. 
Because the issue of standardized outcome measures is so salient to vocational 
education, I believe that it would serve a useful purpose to analyze the need and use of 
standardized outcome measures from the triple perspective of the federal, state and 
local levels. 

The impetus from the use of standardized outcome measures emanates from federa! 
vocational education legislation. Congress, in enacting the Education Amendments of 
1976 Title II stipulated that a vocational education reporting and accounting system be 
estabJished! This data system, commonly referred to as VEDS, is to be based on uniform 
dfefinitions and is to contain elements descriptive of vocational education students, 
programs program completers and leavers, staff, facilities and expenditures. Examina- 
tion of House and Senate reports accompanying the legislation reveals that VEDS was 
established to overcome the lack of adequate data describing the vocational education 
enterprise in this country. A common complaint echoed in the testimony was that 
vocationat education data were not compf-i^-^'o across states and hence that 
aqgregatlons at the federal level were of questionable validity as indicators of the status 
of vocational education. This situation was frequently described by the adage that "you 
can't mix apples and oranges and get anything but fruit salad." Given the pervasiveness 
of the problem and the harmony of voices raised in protest of the lack of adequate data. 
Congress responded by mandating the establishment of a nationai:y uniform data 
reporting and accounting system in vocational education. 

In addition to VEDS. Congress Introduced further standardizations by requiring that all 
state vocational education programs purporting to impart entry level job skills and 
receiving assistance under the Act be evaluated according to the extent that program 
completers and leavers find employment in occupations related to their training and are 
considered by their employers to be well trained and prepared for employment. This 
requirement, coupled with the requirement that evaluation data be included as part of 
the VEDS, made standardized outcome measures in vocational education a reality. 
Given the' existence of this reality, I would like to spend some time tracing the 
implications for vocational education. In order to do so, it is necessary to examine the 
effects of standardized outcome measures as they impact across federal, state and local 
levels. 
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The Federal Perspective 

Melvin Barlow noted in his bicentennial review of vocationaPeducation that the greatest 
influence on vocational education has been the Congress^ This influence is currently 
focused on the use of information in vocational education to improve rationai data 
based planning. The philosophy underlying the recently enacted legislation is that better 
data will result in bettor decisions, and that better decisions will ultimately improve the 
quality of vocationaheducation. Concurrent with their emphasis on the need for 
information to support better decision making is the requirement for information to 
monitor the progress of yocational education to ensure that the intent of the iegislation 
Is being carried out. It is these two themes, the need for information to support 
improved decision making and the need for information to support monitoring of the 
status of vocational education and their complex interplay that provides the 
Congressional rationale for standardization of outcome measures. 

Congress, in carrying out an expanded monitoring role., provided for the flow of 
standardized outcome data from the locals, through the state to the federal level This 
information is to be provided by VEDS to the Commissioner of Education who is 
required to submit an annual report to Congress on the status of vocational education. 
This report is to contain an analysis of the data, presumabiy to determine its policy 
implications, and is specifically mandated to contain a summarization of the outcomes 
of vocational education as measured by the standardized outcome data. So as to have 
an independent check on the status of vocational education. Congress also provided 
that the National Institute of Education undertake a thorough evaluation and study of 
vocational education at state and local levels and report its findings to thi president and 
to the Congress no later than September 30, 1980. One can but surmise that these data 
will be used for cohtinulng oversight of vocational education. It is quite conceivable that 
Congress will publish an oversight report on the implementation of the Educational 
Amendments of 1976, Certainly the increased avaitr'^ '^ty of standardized outcome data 
cannot help but influence Congressional opinion ai lo the ability of vocational 
education to respond to prevailing economic and social needs. The valence of this 
opinion will have its Impact on subsequent Congressional appropriations and will 
undoubtedly shape the format of federal vocational education legislation. The effects of 
standardized outcome measures will extend beyond ihe halls of Congress. The 
administration position with respect to vocational education could easily depend upon 
the image of vocational education as portrayed m the ar nual status reports. The result 
of their percent' i might well be reflected in the administrative budget with allocations^ 
to vocationai nation conditioned by the administration view of vocational education's 
ability to impaci on significant social and economic problems of immediate political 
interest. 

Advocacy of vocational education at the national level would be facilitated by the 
availability of standardized outcome measures. Professional organizations like the 
American Vocational Association would have access to information documenting the 
progress of vocational education and the extent of unmet needs remaining to be served. 
Given the credibility of the process used to generate these data, this capability to 
document progress and needs might increase vocational education's competitive 
position with respect to the share of federal funds received. The same positive results 
might accrue to the National Advisory Council, Availability of a pool of standardized 
output information should provide them with a data base for the monitoring of the 
progress of vocational education in meeting the national manpower needs. 
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While offering the possibility of positive effects at a national lev^^l, uniform data on the 
quality of vocational education might have some uniform consequences. Provision of a 
single data base will result in all agencies using essentially the same source of 
information on which to base their decisions. The possibility of al! actors at the national 
scene having access to a common data base poses some interesting questions For 
example, 'Will the advantage go to those who have the capability to make the most 
astute analysis now that data present no clear cut evidence as to underlying causal 
factors. The sam,e data lend themselves to multiple interpretations with differential 
policy implications. Given this to be the case, one might rightly be concerned as to 
whether the advantage will go to those who are most astute in the use of data to 
buttress arguments that are supportive of their position. Since agencies would be using 
the same intelligence system, agencies like the National Advisory Council might find it 
difficult to maintain an independent and impartial vantage point. Efforts to acquire 
independent data for purposes of veriffcation would be so limited m comparison with 
the size of the national system that its utility for verification purposes would be 
questionable. 

Use and ultimate utility of a standardized vocational education data system v.HI depend 
on whether this system is primarily accounting or decision=oriented. An accounting 
orientation will predispose the collection and reporting of standardized indicators of the 
status of vocational education. These indicators would be periodically released in the 
form of reports with fixed format and content. The pijrposes would be to provide 
descriptive information on vocational education students, programs, expenditures, staff 
and outcomes as measured by follow-up studies of completers and leavers. The 
principal utility would be to provide baseline information on the progress of vocational 
education. Since data elements would theoretically be based on uniform definitions and 
standardized collection procedures, data could be aggregated at the national level, 
thereby avoiding the traditional ^apples and oranges' problem. Vocational education 
status indicators would be similar in concept to the national labor market indicators 
collected by the Bureau of Labor Statistics and demographic data collected and 
reported by the Bureau of the Census, 

In contrast, the decision^oriented approach would emphasize the analysis of standardized 
data to support policy decisions. Rather than providing a static accounting of the progress of 
vocational education* a decision-oriented system would be structured so that the data could 
support theJdentification. formulation, and choice of decision alte natives. The emphasis 
would be more on the use of historical data as a basis for anticipating future trends and the 
analysis of data to support testing of hypotheses about the causal factors interacting to 
product the observed results. Because of the need for data to support decision making, 
greater flexibility in output format would be required, with implications for data storage and 
retrieval. The data system would have to be structured to allow greater accessibility to the 
data files on an as needed basis. 

The development of a standardized vocational education data system will both shape 
and be shaped by the context In which it evolves. The accounting orientation has the 
support of precedent and Is congruent with the accountability thrust of the present 
legislation. One would anticipate that an accounting-oriented vocational data system 
would place primary emphasis on the production of vocational education statistics 
descriptive of the condition of vocational education. On the other hand, one would 
anticipate that a declsion'Onented data system should be capable of providing 
information that enhances the quality of administration of vocational education at the 
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federal leveL More specifically, one might argue that improved access to standardized 
data should have utility in occupational and manpower planning, identification of issues 
and problems of national significance requiring applied research and development, 
sharpened perspectives on forthcoming issues and challenges racing vocational 
education, and improved integration of vocational education into a comprehensiye 
manpov\/er delivery system. 

The most critical consideration^governing the use of standardized data at the federai 
level will be whether data are used primarily to ensure compliance or to support 
improved leadership. A choice for compliance will resuli irtthe data being used primarily 
to monitor the performance of vocational education for the purposes of accountability. 
Emphasis on the decision-making orientation will result in data being used to support a 
stronger advocacy position for vocational education. 

The use of data at the federal level is constrained by the nature of the federaUstate 
relationship. The constitutional separation of powers places responsibility for education 
at the state leveL Consequently the federal role in the federal-state partnership, as 
'historically detmed, has taken the shape of federal financial aid to the states to assist 
them in dealing with problems of national concern as defined by Congress. The states, 
in principle, are free to establish their goals. Once goals are established, it is the federai 
responsibiiity to determine that means chosen tp achieve these goals are in accordance 
with legislative intent. This partnership relationship by its very nature is conducive to 
use of data as a measure of compiiance. As a result of the emphasis on accountability, 
comtemporary data systems have been primarily accounting type data systems. 



Because the constitutional authority for education resides with the states, Congress has 
assumed that the responsibility for vocational education also rests with the states. As 
exemplified in the Educational Amendrrients of 1976, the state is cast in the role of a 
master planner. The state plan as the master blueprint, in accordance with the principles 
of rational planning, is to contain a statement of the need for Job skills within the state, 
the goals that the state will seek to achieve in satisfying these job skills, and a 
description of the programs and courses to be offered to achieve these goals, in 
addition, the plan is to include a description of the use of federal, state and local 
vocational education funds for the achievement of these stated purposes. 

The legislation draws no distinction between the state's role in goals planning and in its 
role in program planning. Goals planning is concerned with the determination of what 
needs exist to be fulfilled. Program planning, in contrast, refers to the way that 
resources are combined and marshalled to serve identified needs. In many instances, 
states have delegated responsibility for program planning to the local leveL The result is 
that although the state may be able to identify goals, they do not have direct control 
ovf r the delivery system to achieve these goals, the reason being that in most instances 
the responsibility for the ultimate delivery of vocational education programs resides at 
the local level 



Although trife states have constitutional authority for education, they have in effect 
delegated this authority to the local leveL As a result, the state is in much the same 
position with respect to the locals that the federal government is in with respect to the 



The State Role 




spates The effect is that state use of vocational education data has evolved mamly as an 
accounting function in response to federal reporting requirements^ Since relativoly few 
states are in ? position, nor would they wish to impose the state will on the locals, tncrt- 
has been little need for information systems to develop to support rational centralized 
program planning. As a corollary, there has been relatively little need for state systems 
to include output measures of local performance. Because of state law, state board of 
education policies, a.id the relationship of vocational education to general education, 
there Vs relatively little discretionary authority over the flow of federal and state funds to 
locals, 

The disparity between the de jure structure of the legislation and the de facto structure 
of reality is exemplified in the difference between state and statewide needs State need 
connotes that a need is defined by the state agency by virtue of its centralized authority 
Statewide need, in contrast, connotes a need that is pervasive throughout all the local 
units comprising me state as a yuQ-poiiiicai enuij w.^.^w.. ,,m .■■<<^- . , ^ 
operationally defined as a need experienced by a maionty of locals. Tnis definition 
tends to shift responsibility for need determination from the state to the local level 



The Local Scene 



Decision making at the local level is frequently influenced mere oy poiincai 
considerations than the need for data to drive rational planning models. Need for 
prourams is often based on community rather than statewide considerations. Local 
advisory councils represent a widely used means of gathering information about local 
n«eds and concerns. Other sources of community input include school board members 
who are generally influential community citizens, community pressures focused on the 
superintendent and other school administrators, and the community linkages maintained 
by vocational education teachers in the conduct of their courses. 

A host of associated factors mitigate against rational data-based planning as interpreted 
by planning scholars. Decisions at the local level are frequently constrained by scarce 
resources Since state monies tend to flow according to enrollment statistics, 
administrators are reluctant to drop courses that are popular and hence paying their 
way simply on the basis of outcome statistics showing placement rates in occupations 
related to training. Educational resources are frequently not easily transformed into 
other uses. Staff/equipment and facilities, once acquired to support particular program 
offerings are not easily shifted to accomodate newly emerging demands. The scarcity 
and frequently limited substitutability of resources tend to restrict the discretion of local 
decision makers and thereby reduces the utility of data to contribute to improved 
program planning. 

Because the accounting (indicator) data frequently required for reporting purposes are 
of limited use in local level decision making, the collection and reporting of such data 
are laraely on overhead cost. Data collection and reporting efforts are geared to 
minimize this overhead cost with consequent effects on the quality of the data reported. 
Since many of the locals understand the economics of this trade-off and its 
consequences on the quality of data reported, they express little or no confidence m 
higher level aggregations of this data. 
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Although this discussion pertains to data systems in general, the application to 
standardized outcome elements is immediate. Local schools with established vocational 
proorams linked into the community infrn^structure, are not aomg to be dissuaded to 
st=:r offering a program simply on the basis of output data showing low placement ratps 
m occupations judged by an external standard to be related to the^program. The 
program may. for example, be a popular program with the student. The instructor may 
be well liked by both students and crmmunity and the program may be perceived by th^^ 
community to be serving a useful community function. A prime example of this 
phenomena is vocational agriculture. By a strict interpretation of production agriculture, 
placempnt rates might be low for a particular program, yet the program may have 
widespread community support in rural communities. 

Statistical data that are inconsistent with the nexus of information regarding program 
support will tend to be neutralized by challenging the validity of the data. This challenne 
can take place on a number of grounds. A likely basis of challenge would be on the 
definitions and values underlying the data. With lespect to standardized outcome 
measures, this challenge would focus on the validity and acceptability of the concepts 
underlying the data source. Criticism of placement rates would likely take the form of 
criticism of the definition of related occupations and a challenge as to the efficacy of 
placements as the sole indicator of the value of vocational education. 

Again, referring to the vocational agriculture example, the validity of the data might be 
challenged on the grounds that production agriculture represents but a small fraction of 
the occupations that require the skills developed in vocational agriculture programs. 
This argument might also be augmented by the contention that vocational agriculture 
teaches a philosophy and a way of life that is urgent for the preservation of the values 
and morality of rural America, The negative effects of low placement rates might be 
further countered by the argument that vocational education prepares people'for work 
rather than for specific jobs. Using this reasoning, the data could be said to provide 
misleading results in that they did not depict the number of vocational students who 
created their own employment because of skills learned in vocational programs, nor did 
they account for the contributions made by vocational education to the subsequent 
career achievements of vocational students. 

The purpose of these examples is to illustrate that the use of outcome data is largely a 
matter of attitude. If the data tend to be in accord with prevailing attitudes, the resufts 
will tend to be accepted as a rationalization of the validity of the foundation values and 
beliefs. It on the other hand, the data are at cross purposes with these fundamental 
values and beliefs, the validity of the data will be challenged In an effort to reduce the 
dissonance created by the incongruent piece of evidence. Those of you acquainted with 
psychology will recognize this as an example of coiinitive dissonance theory which 
states that a person will act to reduce the dissonance created when information conflicts 
with vfJues. Basic values and beliefs change slowly and only after data have repeatedly 
shown that these beliefs and values are inconsistent with reality or lead to actions with 
adverse consequences. 



State-Local Relations 



States are often reluctant to take punitive action on what may be interpreted as adverse 
outcome data. For one thing, many state vocational educators share the same 
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apprehension and reservations about the ability of outcome data to capture the essence 
of quality, They are also astute enough to realize that neither the state nor lie faderal 
government can mandate quality programs. Since many of the state supervisors and 
consultanti'^ave carefully nurtured a network of relationships with local personnel, they 
are unwilling to jeopardize the continuance of these relationships by taking precipitous 
action on the basis of information contained in standardized outcome data. They realize 
that whereas they may be able to force a small change in program design and 
operation, a show of force would ultimately reduce their effectiveness as facilitators of 
long-run program improvement. State level vocational education professionals also in 
the main realize that positive change is a slow evolutionary process, State input, if it is 
to effect change at the local level, must first be accepted at the local level. 

This acceptanc^depends upon the extenfto which the information is regarded as 
credible and the^ource, legitimate. The process of establishing credibility at the local 
level is slow to build since it depends upon a relationship of mutual trust. The real 
business of faclTTtatIng programs is done on a personal basis through informal 
relationships. Thus, the formal data flow Is often a matter of formality with decisions 
having bean reached by common consensus. In this personal relationship, the state 
consensus partners generally respect thesovereignty of the local to be In the best position to 
know its local needs and conditions. 

The formal-informal relationships between state and local are paralleled by the flow of 
information. Formal information tends to be that required as a matter of compliance and 
is generally imposed by federal and state lawc and administrative policy. Whereas the 
formal Information structure supplies the data for reporting requiremenfs, the informal 
Information structure generates much of the input into the decision-making system. As a 
result, the formal structure acts as a pipeline or conduit of Information for data reporting 
purposes that flows upward through the state and ultimately Into the sea of federal 
, information. Because this flow tends to be isolated from the info<^mai system, this 
information has relatively less impact on decision making and serves mainly for 
satisfying compliance purposes. The state in this process functions ^much as a 
centralized warehouse, collecting Information from the locals, packaging it, and 
forwarding it on to the federal destination. 

Whatever problems existed at the state level with respect to mandating quality are 
magnified manyfold at the federal level. The federal level, being further removed from 
where the action is, has to rely on the data generated by the formal system for 
information as to the current status of vocational education. Lack of informal 
information makes the federal level dependent upon formal inforr nation for intelligence 
with the result that tRe data, Incl^uding-stantfardized outcome data, present a rather 
cloudy, incomplete, and freqtrwnlly inaccurate picture of vocational education. 

Suggestions for Improved Utilization 

Given the structure of vocational education and the impact that this structure has on 
Information, the question of immediate concern is what can be done, if anything, to 
Increase the effective utilization of information by vocational education decision makers. 
To say that I have the answers to these questions would be presumptuous, I can only 
share with you some views and notions as to how the process n'light be modified with 
the hope of improved data utilization, 
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I believe it imperative that we distinguish between the use of data for reporting purnoses and 
the use of data for program improvement. In recognition of this distinction, I prop jse that we 
find a small core of standardized data elements that will serve to indicate the status of 
vocational education. The data elements to be collected should be Judiciously chosen so as 
to present the maximal amount of information to the major users of vocational education 
data at the national level. The dimensions of this core of data elements should span the six 
categories provided for in the legisiation. Specific elements within each of the six categories 
could be chosen so as to answer the most frequently asked questions of vocational educa- 
tion. Since Congress Is the major consumer of vocational education information at the 
national leveL it would seem appropriate to use [heir informational needs as criteria in 
selecting the data element composition for the core of indicator items. 

The temptation to expand the core of Indicator variables collected should be resisted 
lest the burden of collecting this Information on a regular basis become too burdensome 
and costly for the benefit accrued. Because no data core regardless of its size c^n 
answer all the questions that might be raised, provision should be made for rapid survey 
capability to secure answers to questions, providing that there is sufficient need. One 
salient indicator of need would be the willingness of Congress or a federal agency to 
appropriate money for the conduct of the survey. 

To insure comparability of the indicator measures, their format should be rigorously 
controlled. This could be achieved by uniform definitions and standardization of the 
collection procedures. Development of uniform definitions and standardized collection 
procedures should be under the auspices of the agency having responsibility for the 
coHection of indicator data/ Since the National Center for Education Statistics has 
legislative responsibility for the education data function, this agency would be a logical 
choice. The elements should be operationally defined and the collection procedures 
based on a sound technical and itatistical methodology. 

The data elements should be structured so as to permit rapid retrieval of information in 
a flexible format as defined by the user. The flexible format would circumvent the 
present difficulty of Information being available only in the tabular report forms chosen 
by the report writers updated only as often as the agency publication schedule permits. 
Those of you familiar with federal report publications know that this updating often 
entails a considerable time lag= 

The role of the state and locals in this collection process varies considerably. One 
option would be to use Arthur Lee's idea and have the locals submit data on individual 
student records in machine readable form. This would circumvent the rather 
cumbersome current process and would eliminate the laborious activities associated 
with filling out current reporting forms. However, this assumes that most local schools 
have data processing equipment. Another possible limitation is that the sheer number of 
records involved would exceed the processing capacity of a centralized system. 

An alternative to securing information directly from the locals would-be to' use the state 
system as an Intermediary. State systems could be used to preprocess the Information 
obtained from the locals and could send the processed data to the national level in 
machine readable form. Since most states have automated data processing capabilities 
of some sort, this would overcome the equipment limitations associated with having the 
locals directly involved. The states might also exercise some preliminary processing 
such as reliability and validity checks on the data to ensure that they are in proper form. 
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By decentralizing some of the responsibility to the state level, the states could be more 
involved In the process and the burden imposed on the central processing unit could be 
reduced. In this sense, the system cotild function much as a distributed computer 
network. 

So far I have not touched on how the reporting burden may be reduced. This could be 
accomplished in a variety of ways. First, the reporting burden could be reduced by 
restricting the size of the common core of indicator variables requested. Locals might be 
paid to provide the Information according to federal reporting standards. This would 
make for a more equitable relationship since if the federal government were paying for 
the data they certainly could expect to have It reported according to their specifications. 
Another option would be to pay an independent agency to sample data from the schools 
much like the Census Bureau now secures current population data from samples of 
households. It Is conceivable that the state agency might be paid a stipulated amount to 
secure this information on a specified sample of units according to the standardized 
procedures. 

The major question raised by the critics would be that of cost, r maintain that the cost 
would be no greater and might result in a savings. Indicator data collected on a 
well-drawn sample could provide data of sufficient precision for federal purposes at 
significantly less cost than the universal sampling currently being used. The fact that 
schools would be paid for their effort expended in collection of data according to 
externally defined standards would certainly do much to improve the quality of the data 
collected. The major difference in this proposed method and that currently being used is 
that the cost of GOllectIng data would be directly borne by the federal government rather 
than indirectly shifted to the states and locals. Funding for this data collection effort 
could be achieved by several methods. Congress might provide a special appropriation 
to NCES for performing this function. Another option would be to transfer a stipulated 
amount of vocational education appropriations to NCES, or a thjfd\option might be to 
stipulate that the states use funds authorized for planning unddr Section 102(d) for 
purposes of data reporting. 

The effect of these recommendations would be to test the utility of data indicating the 
status of vocational education. If these data have utility at the national level, then 
Congress should be willing to pay for the collection of these data in the same way that 
It brovides for the collection of unemployment and employment data and census 
Inpicators. If there are no advocates for the collection of data at the national level, then 
it is not cost effective to collect and should be discontinued, 

Tp say that data only have use for reporting purposes would be a pessimistic prognosis 
for the future of vocational education. I am optimistic that the quality of vocational 
education decision making can be perfected by the provision of improved data. 
Furthermore, I believe that the route to significant improvement Is through the medium 
of technical assistance. The focus on assistance is consistent with the purpose of the 
1976 Amendments and provides a more positive approach to improvement than that 
provided by a compliance emphasis. I believe that if the federal state-local partnership is 
to more thar rhetoric, then each partner must bring something to that partnership. 

The federal and state contribution to that partnership is assistance provided in a spirit 
and form to augment and strengthen local operations. The local role in this partnership 
is to accept assistance in the spirit in which it Is offered and to rise to the challenge of a 
quest for continued improvement. 



39 



A national data base of standardized indicator variables could provide the mechahism to 
support the provision of technical assistance at the federal level. Three principal 
agenoies could be coopeTatively involved in the assistance effort. The Bureau of 
Occupational and Adult Education (BOAE) has legislative authority for the administra- 
tion of vocational educatldri legislation. The National Center ©f Educatidn Statistics 
(NCES) has legislative authority fo^the development and operation of/a national 
vocational education data system. The iNdtional Occupational Information Coordinating 
Committee (NOICC) has responsibility for the design and implementation of an 
occupational Information system to serve the national, state and local needs for 
occupational related information. Working in close conjunction, these agencies could 
exert a powerful force for the improvement of vocational education through the 
increased utilization of standardized data. Analysis of the national indicator data to 
determine the implications for vocational education would provide an Informational 
basis for the proviblon of technical assistance. 

Capability within the Bureau of Adult and Occupational Education to analyze data for its 
policy and management Implications would contribute significantly to the enrichment of 
BOAE leadership functions. Exernplary data capabilities might be the identification and ■ 
interpretation of emerging trendf in the demand for and the support of vocational 
education. Enrollment data couM be analyzed for shifting patterns and the factors 
associated with these patterns identified. Similar analyses could be conducted with 
respect to vocational education completers and leavers with a view toward identifying 
the indi^vidual and programmatic factors that contributed to successful vocational 
^ education outcomes as determined by a variety of criteria. Insights into the factors 
associated with vocational education performance as measured by the standardized 
outcom^indicators would serve as an objective basis for identification of problems of 
. national significance. Such data offered to the states in the spirit of assistance could 
constitute a salient force for change. By providing the state assistance in the 
interpretation of the^data and its significance for vocational education, state and local 
levels would see some possible benefits accruing from the data that they had provided. 
The technical assistance role also Includes NCES. A principal role for NCES would be 
Ho play the lead agency role in the development, operation, and updating of a policy 
relevant vocational education data system. Policy relevance would at the minimum, 
require analytic capability In the system to support the determination of functional 
relationships between data elements. This would require data based management and 
support of statistical analysis nrocedures. Policy assistance to other agencies might take 
the form of simplified computing routines and/or the provision of personnel assistance 
in analysis and interpretation. Technical assistance could be provided to the states in 
order to facilitate their use of data to support administrative decision making. This 
assistance might take the form of suggested analytical procedures that the states could 
use in analyzing their state level data bases, alternative methodologies that the state 
might wish to employ to augment the indicator data currently being collected and 
reported nationally, and training packages and conferences designed to facilitate the 
understanding and use of reported data. Methodological assistance in the collection and 
analysis of data at the state level would serve to stimulate the demand for more and 
improved data. The important consideration Is that the demand would emanate from the 
users' need for data to support internal operations rather than to satisfy external 
reporting requirements. 
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The occupational information system to be developed by the National Occupational 
Information Coordinating Committee (NOiCC) is yet another potential mechanism for 
stimulating the use of data in response to internal needs. Whereas NOICC is to provide 
the design standards for the development of an occupational information syBtem, the 
responsibility for Implementing this System at the state level rests with the State 
Occupational Information Coordinating Committees (SOICC), The relationship between 
the NOICC and the network of SOICCs provides yet another mechanism for technical 
assistance. The NOICC, in addition to providing standards for the design of an 
opcupatiorTal information system, could assist the states In Interfacing and drawing 
together the large complex data producing systems in the state into an Integrated 
informatlbnal network. Occupational demand data could be more effectively linked with 
supply, and a more integrated picture of supply could be provided across educational 
levels and agencies concerned with human resource development. The requirement of 
standardization of definitions across systems could provide a vehicle for the linkage of 
these systems at the state level Into a more cohesive and comprehensive Informational 
system. By linking the existing data systems in the state into a confederation of systems 
based on shared definitions, It should be possible to shape the information Into a variety 
of formats to suit the needs of various user groups. By serving as a vehicle to promote 
dialogue between various user groups and the producers of data, the SOICCs can 
perform a useful function in facilitating improved understanding of data that Is currently 
available, the methodology underlying its collection, and the potential uses to which this 
data might be put. NOICC could fulfill a useful function by providing technical 
guidelines and assistance to the SOICCs in organizing and focusing state efforts on the 
implementation of a unified occupational information system. This assistance could take 
the form of technical manuals designed to facilitate state understanding of the 
procedures underlying the collectloh of certain data elements, assistance in performing 
needs assess fpn en ts to determine the informational needs of various user groups at state 
and local levels, provision of Information descriptive of the alternative methodology for 
collecting information identified as being needed, and stimulation of state interest by 
providing financial support for the collection of Information that might satisfy a need 
common to a variety of users. One exampi^e of such an area might be that of a statewide 
survey to determine the universe of need for education and manpower programs, 
NOICC could promote this activity by providing funds and technical guidelines lo the 
SOICCs to assist them in coordinating thi^ activity within their respective states, 

\ \ 

State occupational information coordinating cdmrriittes could become the nucleus for 
the promotion of the use of data for the Improvement of education and training services. 
For one thing, the explicit purpose of the SOICCs Is to improve the dialogue and 
cooperation existent between agencies involved in the delivery of education and 
manpower services. A concern for occupational Information is the common basis and 
provides the rationale for SOICC organization. Since SOICC exists as a vehicle to focus 
coordlnative attention on the need for and the use of occupational Information as a 
means of securing improved program planning, each state for tha first time has a staff 
committed to the promotion of the development and use of a coordlnative data base. 
Thus, the SOICCs have the potential to launch a movement to more actively Involve a 
wide constituency of data users and to develop support for increased involvement in the 
data production process. 

In order for this movement to achieve its fullest potential it Is imperative that the 
concept of cooperation inherent in the SOICC structure be extended to the local level. 
The extension of a state level network of information users and producers can be 
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extended into the regions by the establishment of a network of regional information 
centers one center to be established for each region. The major purpose of the regional 
information centers would be to function as the hub of a regional information system. 
The major purpose of a regional information system would be to interject the 
uniqueness of local labor market conditions into a comprehensive statewide information 
system. Functions of the regional information system would be both to collect and 
provide occupational career and educational and manpower information pertinent to the 
region served by the center Regional information centers might provide information on 
current and projected occupational employment for that region, current job vacancies in 
the region, demographic Information pertinent to the region, information on current and 
anticipated economic conditions, information on the availability of training and 
education opportunities of the region and evaluative data gained from follow-up and 
placement of the clients of CETA and vocational education programs as well as other 
programs that might eventually carry out follow-up activity. 

Regional information centers might also be repositories for information (from a variety 
of sources) that pertains to the social and economic characteristics of the region. With 
regard to career information functions, regional information centers could provide 
information and referral services to people concerned about the availability of education 
and training opportunities in the region. The centers could also provide information on 
available assistance in the region for job placement, counseling and guidance services 
or for other programs designed to prepare and assist people in finding suitable 
employment. Additionally, they could provide information about duties, requirements, 
wages, and employment prospects for a variety of occupations to be found in the 
region. 

These regional information centers could also supply information to a host of local 
community organizations. Examples of organizations redeiving input from regional 
information centers might be occupational counseling and guidance centers at both 
secondary and postsecondary institutions, local and industrial development commis- 
sions, planning officers of educatiohal and CETA agencies, local community education 
and work councils, community action agencies, county and metropolitan planning 
officers and a variety of citizen action groups. Data collection activities of regional 
Information centers might include: collection of information from employers as to Job 
vacancies, characteristics of workers customarily hired, includlr skills necessary to 
perform the job, required personal qualifications, training oppor nities, and hiring 
requirements. Becpueo of the involvement of local repr^^ tivei in the operation of the 
canter, the likelihood of employer participation and pre , jn of information would be 
greatly enhanced due to increased rate of return for the employer both in terms of 
better trained employees and the public relations accruing from participating in a 
community based activity. 

A regional Information system could also contain information that could be used to 
assist in the planning of vocational education and manpower programs to serve the 
region. Program planning information might include an inventory of education and 
training opportunities in the region provided by vocational/manpower program delivery 
systems and training programs provided by private employers. Regional information 
centers may provide a technical assistance function through the provision of a wide 
variety of information. This information might include legislation both federal and state 
pertaining to the development of resources at state and local levels, federal and state 
rules and regulations that might have an Impact on ' planning of education and 
manpower services and reports and other documei r .suits of research and 
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development efforts of other states and communities in dealing with the problems of 
developing human resources. 

These regional Information centers could serve much as public libraries with information 
available upon request. Information from the local vocational education data system 
maintained by CETA prime sponsors, and data systems maintained by local employment 
security agencies could be provided to the regional information centers. This would 
make information readily available that would provide public knowledge of the 
effectiveness of education and manpower service delivery programs. 

Thus, I have come full circle. Whereas the locals are the originators of data describing 
the outcomes of vocational education, they must also be the final users of this 
information If any benefit from them is to result in the ultimate improvement of 
vocational education. The proposed system provides this feedback link under the 
general rubric of technical assistance— flowing from the federal to the stato and finally 
through the regions to its ultimate use in the decisions that shape the process and 
ultimately the final outcomes. 

The challenges are great. However, I believe that the time for action is now. Legislation 
has created a mandate for the improved use of information and has created a variety of 
mechanisms to support this Improved use. Whether history will ultimately record these 
mechanisms as yet another futile attempt, or whether they will provide the means to 
. move us to a new plateau depends upon our foresight, ingenuity and imagination in 
structuring these mechanisms to serve our needs, I am hopeful that vocational 
education will play a lead role In these pitaneering efforts. Although the challenges are 
great, I believe that vocational education has the vitality, the ingenuity, and the creativity 
to overcorne these obstacles and to move to greater achievements in the promotion of 
human well-being. 

{ ^ — ^ ^ 
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CBITERIA AGAINST WHICH VOCATIONAL EDUCATION 
SHOULD BE HELD ACCOUNTABLE 



Grant Venn 



The long recognized success of American education has caused most educators and 
citizens to rely on accountability criteria with which they are familiar, even though the 
future which the learner must face today is far different than it was in the past. It is also 
true that the skills, knowledge and experience needed in the work world of tomorrow 
require more and different competencies than were adequate in the past. 

Recant studies by the National Academy of Sciences titled Assessing VocBtional 
Education Research and Development; by Project Baseline at Northern Arizona 
University titled Report to the Nation on Vocational Education 1975; by the Panel on 
youth of the President's Science Advisory Committee called Youth: Transition To 
Adulthood; by the National f^anpower Institute of Washington, D.C,; in a book called 
The Boundless Resource: A Prospectus for an Education/Work Policy arid others related 
to the problems, issues and priorities of preparing individuals for a future work life all 
indicate the need for new criteria against which to evaluate education of all kinds and 
especially vocational education. 

Traditionally* criteria tor accountability In regard to the preparation of youth for the 
future, iricluding the work role* have been determined over a long period of time based 
on experience, wisdom and judgment of the society power structure. This method has 
worked well» primarily because the future was like the past and an understanding of the 
past was the best way to determine what the future would require. The problem of "what 
to measure" was relatively simple as compared to today. 

It was even more lil/ely that the criteria of accountability for vocational education was 
relatively simple wilh work and jobs remaining the same over generations and for most 
people being primarily manual. The same could be said for nearly all of education in 
terms of being static. Thus, the measurement of success in meeting these rather simple 
and static criteria became the isolated effort of a few professionals and a few dollars 
and was not part of the main concern of education, nor considered highly significant by 
most educators, citizens, or policy makers. The amount of money spent and past efforts 
indicate the degree to which tho selection of criteria against which vocational education 
was to be held accountable was not a significant evaluation priority. 





Thus, a concern for methodology for measurement became more significant than the 
selection Of criteria against which to measure. Without even implying any ciiticism, it is 
fair to say that those in the field of research and evaluation found themselves forced to 
deal with short term, small parts of the education process rather than broad policy, 
direction or criteria setting, or even the broad question of accountability criteria. 

The long and continued isolation of education from the rest of society's daily pressures 
and conflicts, especially the education work relationship, forced the practitioner to 
select criteria for success in vocational education (placement in a training related job) 
that was not of a great concern to the rest of education and specifically nor to 
specialists in the field of educational research and evaluation. 

Very little vocational education research, development or evaluation exists outside the 
overall educational research and development effort in the country. The history of 
research and development efforts at the federal leve! in the United States Office of 
Education illustrates the point. Today, direct work and Job related efforts are probably 
more clearly identified with the Department of Labor than with the Office of Education. 
Even the most recent reorganization of the National Institute for Education (NIE) points 
again to the fact that for whatever reason the relationship between educational research, 
evaluation and measurement and success criteria in vocational education has been 
tenuous at best and Is still insignificant in research area priorities, which is suggested 
by the NIE reorganization. 

It is also true that the dominance of physical science research methodology, where 
variables can be controlled and outcomes today are as they were 1000 years ago. often 
causes educational research to keep searching for the ''right way." There is a ''mystique** 
that "good," "solid," and "respectable" efforts presently in use will eventually triumph 
and that the "right criteria" against which vocational education should be held 
accountable are similar to past criteria. The problem is seen as a need to simply gather 
the right data* analyze It properly, and draw conclusions which Wf/I obviously result from 
such evaluation. It is also likely that many practitioners feel that all the fuss about 
accountability and the new legislation compliance requirements are figments of 
somebody's imagination and In no way related to the real world. This does not imply 
that whatever evaluation is done can be done without sound processes. 

However, it may be that the search for evaluative criteria that are nonchangeable may 
become the search that is unending and could lead to the cul de sac that methodology 
has often entered: if we can't measure it, ii isn't signlficanti 

WhUe there can be no certainty that any criterion chosen can stand the test of time, it 
seems certain that present criteria are Inadequate, often conflicting and not clearly 
defined at any level, federal, state or local. The criteria that are often used by evaluators 
are sometimes part of the resistance to creating new organization, policies, and 
objectives that could help. make vocational education more viable, flexible, and relevant 
for the learner, the taxpayer and the employer. The reason for this statement Is the fact 
that as the criteria for accouritability change it will be difficult, if not impossible, to find 
the most valid and reliable methods of measuring success, thus. It becomes important to 
not change the criteria from ^ standpoint of the evaluator. The recent and continuing 
national reaction to falling ^ \ scores indicates the problem. Even though the criteria 
were chosen to predict success in college, and were predicting successfully, many 
persons argued that the high schools must go back to their one purpose preparing 
youth to enter college. Multiple criteria and changing criteria make the job oif evaluating 
success difficult to do and difficult to explain. 
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iven more troublesome are rriultiple processes by\which vocational education ^criteria 
are established. These processes are even more cogiplex than those in the rest of 
education, for the following reasons: \ 

1. Specific outcome measures and operational ^rJteria are set by federal law and 
regulation. This may be the reason why vocational administrators feel their 
most important and time consuming tasks are tn^ir efforts to comply with 
federal and state regulations. 

2r Employees that hire directly from the vocationally e^cated student body have 
specific and. definitive criteria defined by occupation and by Job, 

3. The economic and social needs of the nation may be more directly or 
immediately affected by the success or failure of vocational education both in 
terms of meeting manpower demands and reducing unerriployment and 
emergency expenditures than by most other education programs in the 
schools. \ 

4. The different methods and equipment used in vocational education are often 
irritants, if not direct threats to the rest of the educational est'abnshment, at all 
leveISp and thus criteria for sucoess in vocational education often gets changed 
to reduce these conflicts rather than defined more directly to measure 
vocational success. 

^ 5. Those directly responsible for vocational education policy and objectives are 
often not familiar with either vocational education or the needs or problems of 
the work world and thus tend to support their own concepts of quality. 

6. The historical separation and development of work and education has created 
a chasm which, even today, Is seldom bridged when most people think of 
either education or wr»rk. 

These few examples are given to point up the inherent and difficult problem of 
establishing criteria against which vocational education should be held accountable that 
will be acceptable to the educator the employer, the legislator, the vocational educator 
and, perhaps most important of alls to the researcher and evaluator. 

Allow me, for purposes of this paper, to redesign Glasser*s 'Ten Untenable Assumptions 
of College Instruction/* tp fit the teacher in vocational education whose criteria are set 
by everyone, constantly dhanging, and now measured against a single factor over which 
he/she has no control, the job market, ^ 

Often the vocational teacher is forced to act as if the assumptions listed below were 
true. 

1. The specific job knowledge to be learned by the student will relate to the 
student's future work and career plans even though the career plans of the 
student aren't known. 

2. All students in each course come with equal aptitudes, learriings and 
experience. 
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3. Aptitude is more important than previous achievement and motivation for the 
attainment of skills and knowledge. 

4. All people learn in the same way and take the same time to learn ihe same 
amount of the same thing, 

5. Listening to lectures, reading materials and learning rules are powerful means 
for changing learner behavior. 

6. Students retain knowledge and skills without much review or use, 

7. Grades tell what a student has learned and can do. 

8. The vocational teacher Is^ by virtue of a teacher's authority, able to teach the 
right things well and knows what should be taught, 

9. Vocational education instruction is enough to aach a student how to maKe the 
transition to work and responsibility. 

10. The organization and structure of vocational education as it complies with 
federal and state regulations Is the best system for transmitting knowledge and 
skills to students at the local leveL 

Each of us would quickly realize that not alf vocational education instructors believe the 
assuniptions, yet the environment in which most vocational education is carried on 
the educational system structure, organization, money and priorities all tend to force 
compliance with traditional educational success criteria and ev' 'nation methods, even 
though a look at the future relationships between edUQation and work would suggp^t 
some new criteria against which vocational education should be held acconntab' 
One thing seems certain, setting criteria for accountability and . m measu, ^g >as 
are not cpnsolldatad efforts In vocational education. At the pre^mt lime, the federal govern- 
ment, through legislation, regulations, and categorical funding; the state througn adminis- 
trative program plans and fund albcatlon; and the local units by interest, compliance or 
nonconcern, create the variety of criteria which the Individual school and teacher must meet 
and upon which they are Judged. Yet these all fade Into the background as the individual 
student, parent, teacher or employer comes face to face with the question, 'Ms this person 
able to be successful today and In the future in his/her private, public, and work life?*' This 
concept of accountability may be a more significant basis for attempting to determine a new 
approach to setting criteria because It Is truly the basis upon v/hich each persoh judges the 
worth of asocial organization, an educational institution, appropriation, or specif ic program. 
What is so significant is that change has become so pervasive, especially in the area of work 
and employment, that it Is fair to say that everyone must be educated in order to work or 
society will be forced to care for them. What may be even worse in our culture Is that those 
who do not work have neither worth nor dignity. 

Thus, criteria against which vocational education must be held accountable in the snort 
and long run must come from the indlvtquals in the society into which the student must 
pass and to whom the student must demonstrate that he/she has learned and can learn 
and that what he/she has learned he/she can apply as judged by others than those in 
vocational education, education, or evaluation. 
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Both process and product evaluation will be made regarding vocational education by 
the public, since each person has had experience in education. If this experience has 
been successful, as it has for most '•power structure people," both within and outside of 
education, the degree to which criteria will be ''right/* as seen by these people, will 
relate to their successful experience in education. However, today, youth unemployment 
and isolation; lack of career goals; falling SAT scores; and growing welfare, education 
and crime costs all contribute to people's fears regarding the quality of education and 
the quality of vocational education. 

In searching for new accountability criteria that relate to individual deveropment, 
societal needs and future change, one must consider some of the questions that need to 
be asked In setting priorities for new efforts in evaluating. 

1. What are some different things that need to be done as contrasted to the 
constant question of how to do better what we are already doing? 

2. How can we be more concerned with change rather than right answers when 
we know that much knowledge and skill has a short life in a technological, 
changing work force? 

3. How do we come to value creativity in the development of new methods, 
ornanization, and purposes over compliance in vocational education? 

4. How do we find criteria against which to evaluate vocationa! education that are 
more useful than the traditional criteria of preparation for placement in a job 
related to training? 

5. Can vocational education demonstrate new ways to learn to all of education 
rather trian mimic other education? 

6. Do vocatfonal educators owe first allegiance to the consumer rather than the 
profession, in the area of accountability? 

There are many other questions that could be raised but the sense of the proposition is 
that changes In bur society have become so great, especially as it relates to education 
and work, that new approaches are more important than merely improving or creating 
minor changes in a system based on the past rather than the future. 

Perhaps a brief listing of a few of these changes that support the premise that a major 
new look at criteria for accountability Is more important than simple improvement of 
criteria measurement or minor additions to the list of criteria would prove helpful. 



Fundamtntil Soeiatal Changei that Demand New AQGOuntabllity Critarie 



Odafiges in the Nature of WDik 

Much of what was needed to be a mature working adult regardleBS of talents, interest or 
work opportunities was learned through experience in the home, neighborhood or work 
place; most youth no longer have this opportunity. 
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Changas in Crlt@rh# for Sueaassful AdulthOQd 

The fading work ethic, mobility in the work place, changes in family structure and role 
all imply new ways to prepare for adulthood as part of education. 

i 

Changes in the Nature of Education and Schooling 

One may need to become more a learner through the education process than to become 
a ''learned" person. To complete an education is more myth than reality. 



More Compiex Passages and Transition in Private, Public and Woric Life 

To prepare for a vocation with the expectation that the knowledge and skills will be 
good for life is no more realistic than to expect that one is likely to live in the same 
place or in the same manner ail one's life. The transitions in adult life will require 
different education than did a stable unchanging past, especially in one's work life. 

The Futyre as Different from today 

The most challenging societal change for which vocational education must help prepare 
the Individual Is the fact that we cannot tell exactly what the future will require. What 
accountability criteria are necessary that will predict success when no one can predict 
what will be needed? This has occurred more often In the work area than any other 



Broad Categories of Evaluation Criteria 

In attempting to suggest new criteria for evaluating vocational education, one must 
recognize that there are new fundamental, social, and Individual needs that can best be 
met fay establishing new objectives or criteria against which vocational education should 
be held accountable. The question of how to measure or the methods of evaluation 
should not be considered in' the original search for better criteria. 

If the question of "how to evaluate'' gets into the processs of setting criteria, as do the 
common arguments of "how to meet the objectives," too often critical needs are 
overlooked because present Instruction and organization patterns and evaluation 
techniques were not designed to be responsive to the new objectives. Therefore, this 
paper will not consider the question of Instructional process or evaluation techniques, 
nor wilj it In any way attempt to demonstrate the validity of the new criteria by giving 
examples of the use of the criteria or examples of how and where they have been 
measured. This, it seems to me, Is the Job of the researcher, the evaluator, and the 
Instructional expert whose expertise is in the area of Instruction and measurement. The 
setting of criteria for purposes of instruction and evaluation is primarily a value 
judgment process which must stand the tsst of public acceptance. 

The previous parts of the paper have attempted to describe the reasons for needing new 
criteria In addition to Indicating some of the specific present practices that are setting 
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objectives in vocational education, and the limitations of some of the-evaluMive 

processes that tend to prevent selection of new criteria, 

The selection of the four broad categories of criteria is based on premises thui :is:>ur 
the primary role of education is preparing for the future, 

1. Education has as its major purpose providing spec^MC knowledge and skMls a - 
.well as learning opportunities that will help the individual prepare for thn 
future. This is also the purpose of vocational education. 

2. Education is n Rociatal necessity in preparing for the future and preparation fen 
WDi k htla ljuftriiiib wmmBAr^ for everyone in a technological society 

3. Education must be responsive to the unique differences among mdivid nls nnn 
among groups in order to educate them for future societal roles, mciuci ng 
work, 

4. Education must be responsive to change as a fundamental component of tfic 
future, especially in the area of work, and thus vocational education must 
design objectives which brings change in program structure and purpuau a-. 
well asiin the environment where vocational education takes place. 

5. Even if education meets individual needs, responds to societal needs, and 
meets these criteria efficiently, there is still the question of the value of these 
outcomes if the individual has no opportunity to use his/her education m a 
work role and society, therefore, receives no benefits from Its investment m 
vocational education. 

\ 6. Vocational education has a specific purpose of meeting societal needs in the 
\ aria of developing work skills especially in the areas where new work skills are 
being created, 

7. All learning for the future cannot be done in the school and the emerging 
skills, knowledge, and understandings learned through vocational education 
are often originally developed ana used outside the school, thus the 
involvement of the community becomes fundamental both to learning and also 
to establishing new criteria. 

The broad categories of criteria suggested in the paper derive from the implications of 
societal change and the recognition that the most common program objectives and 
processes were established in vocational education long before change became the 
mosl valid descriptor of the future. They also assume that regardless of how well 
vocational educatlon^can measure and prove that it is meeting its objectives that many 
persons question the objectives or criteria in use. The categories chosen also reflect 
recognition of the fact that change has created much confusion as to just what 
vocational education should be. 

The categories selected provide the common dichotomy of evaluative criteria into 
process and product categories since the two are' necessary components of the teaching 
and learning process. 
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Quality of Instruction, This category was selected because it is fundamental regardless 
of objectives and regardless of process. However, in suggesting specific criteria that 
might be used, the assumption is made that the future will require that evaluation (1) 
should not be a process that excludes individuals from vocational education because of 
their specific weaknesses and (2) can no longer be a measure of quality of instruction 
Quality in education has too long been confused with selectivity and fear of the 
consequences of poor quality instruction has caused inany persons to call for a retur'. 
to guaranteed outcomes through careful selection of students into the program. This 
may be one of the reasons the fede^^l government is now investing over 310 billion in 
CETA and only $600 i^illion in vocf. lal education. 



C. instruction 

Process Criteria for Instruction 

1. The degree to which alternativu niethods are available and used to meet 
individual student interests, aptitudes, and future working conditions. This 
suggests that a single methodology cannot be equally effective among the 
students. 

2. The degree to which time and organization are flexible to maet ihe nB-ds of a 
variety of students. 

3. The degree to which tool learning skills are offered and learned in order that 
the student may be a continuing learner on the job. 

4. The degree to which new and successful instructional proaesses are rewarded 
as contrasted to traditional instruction. 

5. Staff development related to instructional priorities sfiould be more than simply 
more education chosen by the individual staff member. 

6. The degree that learner asse^jsmeni und individualized program planning is in 
use. 

7. The degree to which outside advice from students, parents, employers, and 
citizens is used to redesign instructional processes. 

8. The amount of individualized instruction offered. 



Product Criteria in instruction 

T The degree to which all students achieve entry level competencies m an 
occupational area, 

2, The range ofWill and learning beyond minimum competencies for every 
student. , ) 

3. The attitude of consumers, parents, students, and 'employers as to the quality 
of instruction. 
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4. Measures of the degree to which vocational courses taken meet individLif i 
student career plans, 

5. The degree to which students leave prior to completion to secure employment 
and with minimum occupational competencies. Early learning may be a 
criterion of success. 

6. The degree to which vocational insttuction has created undersianding by the 
student of the need for other and additional learning. 

7. The degree tc which vocational students have knowleJgii about and can 
demonstrate ernnloyabllity skills. 

8. The decree to whjch students may select or be employed for work requiMng 
competencies beyond the occupational study area. 

9. A measure of individual career growth after leaving the program as compared 
to matched students without vocational education. 

10. A measure of students' opinions of ti.3 value of vocational education some 
years later. 

Relevance of Prograin. This aoncept of accountability criteria would measure the qualit^' 
of vocational ed nation against the criteria of judgments by those being taught and the 
actions of those Jho finance and use the product. In the long and sometimes short run. 
persons outside education select evaluative criteria and often this judgment is more 
percaptive than those whose vested interest that tends toward continuance of program 
and process and evaluation. These criteria also use the action? of potential students and 
the actions of the vocational educator as specific Indicators of the relevance of the 
program as related to Individual needs and societal changes in the work area. 

Ralevince of Program 

Process Criteria 

1. Do all students secure related experience necessary for entry job placement? 

2. Are processes established which assess student competencies upon entering 
vocational education? 

3. Have occupational changes in the area over the last few years resulted in 
program drops and changes? 

4. Are special programs, methods, and evaluation in use to assist minorities, 
disadvantaged and handicapped? 

5. Do "power structure" persons serve on advisory committees? 

6. Are students involved in the school program ^nd assigned responsibility as 
they are able^ 
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7. Are student youth groups an important part of the program? Are they 
delegated authority and responsibility? 

8. Are programs planned to inform students, parents, and smployers about 
vocational education? 

9. Are outside peopi'^ with special knowledge used in the instructiGnal program? 

10. Do "power structure''"parents have their children enrolled? 

11. Are students aNowed to fail and overcome thair failures? 

12. Do students cooperate and assist one another in 'he learning process and. if 
so, to what degree? 

13. Do students carry on self-evaluation? 

14. Are periodic reviews made to determine programs that are not relevant? Are 
they dropped? 



Product Criteria 

1. Do employers hire vocational students prior to hiring nonvccational appiicants 
and to what degree? 

2; Do studerits !earn employment seeking skills and demonstrate these skills? 

3. Are vocational staff involved in solving economic, manpower, poverty, and 
vocational related problems in the community? 

4. Do students learn the latest knowledge regarding work, ^^^^ployment. 
advancement, and the requirements of each? 

5. Do students feei more confident, self motivated future oriented, and capable of 
their own self direction as compared to nonvocational students? 

6. Do business peoplcr and parents have more confidence in the vocational 
student succeeding in the work world? 

7. Do students choose the program and create demand or are efforts solely a* 
recruitment? 

8. Do graduates secure additional education to a greater degree than nonvoca- 
tional employees in the same setting? 



Impact of ttia Pragram. The corcept behind this broad criteria is based on the fact that if 
something works or is needed most persons will buy it or to put it another way, changes will 
be made so the new, successful, and valued program can grow. More money will be invested 
and schedules and priorities will change to allow the new program jo try new things. In other 



words, a success brings more success and support. Since vocational education functioniB m 
the environmynt of other education it is necessary to secure some change in that environ- 
ment to be more effective in vocational education. Evidences of such successes are criterion 
which may be lookedforaswaystoevaluatethequalityof vocational education. It isatypeof 
consumer evaluation. 



Impact of the Program 

Process and Product 

1. Is the schedule and organization of the school or are regulations of the board 
of education changing to accommodate new programs and procedures? 

2. is the record of educational equity better as compared to other programs in the 
educational organization? 

3. Are special funds offered to and used by the vocational department to try and 
test new things and new ways? 

4. Do other educators (nonvocational) visit the school to see new programs and 
processes? 

5. Do the media people know about the vocational program and do they tell 
about it? ■ 

6. Are vocational staff used as advisors in programs outside the school; in 
school? 

7. Have ;ate regulations changed to encourage and reward new programs? 

8. Are federal regulations, laws, and appropriations for vocational and vocational 
related programs changing to accommodate the newer thrusts? 

9. Are parents involved In the process of planning their children's education? Do 
they want to and do they feel it is valuable? 

10. Are services being expanded to meet the needs of adults at the request of the 
adults? 

11, Are vocational students respected as students and has the ratio of poor, 
dropout, and other isolated youth tended more toward normal distribution? 

12, Are employers calling on the school? 

13. Are youth volunteering to serve the community as part of the program and do 
agencies and employers request their help? 

14, Do adult attitudes toward new vocational programs, objectives, and processes 
support what has happened? 

15. Are activities planned to impact upon the schools, parents, and community? 
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16. Are new and different individuals and groups being served by vocational 
education? 



Individual Traniltion and Growth. The ultimate criteria for which vocational education 
should be held accountable is what happens tothe learner when he/she makesthetranrution 
to the work world or to the next step whatever it may be. There are many factors other than 
vocational education which may determine the success of the transition but the ultimate 
question is what difference did vocb.ional education make? Tre process of transition in our 
society has become a major probiem for everyone as they move through the passages of life 
in a mobile, changing, complex society. 

If what happens after leaving the program is to be considered as one kind of criteria, 
then ihere must be some process by which the transition is made in such a way that the 
individual gets to use the vocational education fie/she received and perhaps even more 
important that the individual moves into a situation where continued learning can take 
place. The goal of vocational education may be seen as not only preparing for eniry into 
the workplace but developing a learner whose growth in the work v^orld will be self 
actualized by the learning received in vocational education. Furthermore, the economic 
loss to society in unemployment which may have more long range consequences both 
to the individual and to society. These criteria see the job as a means not the end and 
thus raise the question of whether placement in a job for which trained is a good 
criterion against which to evaluate vocational education. It can certainly not be used if 
the average person changes occupations five times during his/her work life. 



Individual Transition and Growth 

Prooass Criteria 

1. Is assistance available for transition to the work world? 

2. Is preparation for making the transition available and are minimum competent 
cies established? 

3, Do parents approve of this service, do employers, do the students? 

4, Is there instruction to help students gain knowledge and skills to avoid the 
failures that 95 percent who lost their jobs have not learned? 

5 Is follow-up of graduates used to change the program and process? 

6. Do employers and community v^ant to have input into educational criteria 
based on their Involvement in transition and follQw-up of students? Do they 
have input? 

7. Are the community citizen and community agencies used in the transition 
process? 



Product Crittria 



1. Do Students* employers, and communfty use the service'? 

2. Do students using this service secure work or employment with greater 
opportunities for utilizing their talents and training? Is a pattern of change in 
this use discernable? 

3. Do vocational students have long range career plans'^ 

4. Is there evidence that vocational school leprrrers do better at finding, receiving 
and advancing in the work world toward career goals? 

5. Do employer attitudes reflect a positi /e difference regarding vocational 
students as compared to nonvocatior al students in regards to ability to get. 
hold, and advance on the job? 

6. Do vocational students have better knowledge of and understanding about 
future work and present workplace conditions, requirements? 

7. Are vocational students better informed as to future labor force trends and 
problems? 

8. Are vocational students more knowledgeable regarding their own talents, 
interests as related to career plans than other students? 



Limitations 

This paper proposes some new criteria that may be more in the realm of "consumer" 
evaluation than ''scientific measurement/' Right or wrong, the matter of whether the 
quality of vocational education can be measured is a hypothetical question, but the 
matter of accountability is a pragmatic, operational matter. It is assumed that researchers 
and cvaluators as well as traditionalists In education all want to find ways to be more 
successfuL In a democracy, and even more in an educational system which has over 25,000 
inriividua! units from K-6 to separate vocational technical colleges, each independent in 
terms of administration and operation, the determination of success will be established by 
the owners and operators of the individual educational units. What is 'Vight" Is even more 
ethereal than what is "successful," 

There is a true and proper conflict between what ought to be and what is and one of the 
limitations of the criteria suggested in this paper is that too much emphasis may be 
placed on the pragmatic. However, the degree to which the citizen will support the 
"theoretically right" Is often based on the perception of how well the researcher and 
evaluator may help solve and assess the solutions of the pragmatic, operational 
problem. 

No operator, separated from the theoretical, can help but become out of touch as does 
the researcher or evaluator isolated from the practitioners. 

In a sense, all of us in education are theorists as education has become isolated from 
society and especially in the work aspect. 
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Most decisions about education are "value judgments'^ and the long term basis for 
vocational education improvement v^ill relate to value judgments rendered by users of 
and participants in the program, as well as the public official who is accountabie fcr 
financing and solving societai problems. 

The present direct relationship between education and work, now required for nearly 
everyone, requires a new look at evaluation methods and prforities in educational researcfi 
This very fact may be a limitation of the proposed criteria, yet the quesiion of what auqnt to 
be should not prevent analysis and assessment of what is. 

Another limitation of the proposed criteria is that they put overt and covert pressure on 
the prautitioner who has had [ittle if any preparation as to knowledge and skills n ^uired 
to accomplish th& goals and the criteria may meet opposition within the educationaf 
community especially among nonvocational colleagues, v^/ho may claim them to be 
irrelevant to ''the true and proper*' goals of education. 

Lastly, most researchers have been prepared to do a different type of analysis than 
csUed for by these criteria. The ability to advance professionally in the research and 
evaluation field is not dependent upon the kind of assessment suggested m this paper. 

Lastly, these criteria are begged, borrowed, and stolen and certainly -.ome afe 
wrong—that certainly is a limitation. Somw are new but only in context. 

Most of you reading this paper Whi be able to suggest other lln nations. 



Realisilc vi. Ideal Criteria 

What is realistic and.what is ideal? In attempting to respond to this often argued 
classification this paper simply defines realistic as criteria that are known, specific, 
accepted, and measurable. Idealistic criteria are those which are not commonly 
accepted as one's responsibility, dimly seen, and variably described, and for which there 
are few recognized measurement processes; however, they are of such a nature that 
most persons believe "someone ought to do something about ihem.'* 

Ideal criteria can be used if they are put in understandable terms and if th^^*.^ is proof of 
positive help for the individual student, employer, and taxpayer. 



Employment in the FItId for Whirh Trained 

This criterion, vvhile appearing to be realistic and relevant, tends to keep vocational 
education tied to the past. The evidence is that Juch a criterion ..an only guarantee that 
vocational education will br ^^n as inefficient since there are too many other factors 
over which vocational edu. iOn has no control that impact upon this situation. 

The factors that determine whether an individual student can get employment in a 
specific field are primarily those which are set outside education. At any one time the 
rate of change in skills^ knowledge and technology is so great that employment in a 
specific field is more dependent on the following: 

1. local, state and national economic conditions 
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2. 



mobility of empioyer and employee 



3. government contract awards 

4. the shut down of production in a related field 

5. dfccovery of a new energy source or a new machine 
6 the quirks of the interviewer 

1\ the weather 

8. the international conflict picture 

9. the wealth of the Ardbs and the price of oil 

10. the changing values or life styles that are '^in" 

11. ot^ars 

This is not to imply that failure to get work in a field for which the student is trained, if 
jobs are available, is not a measure of poor quality in vocational education, but rather it 
is to say that this criterion, when it dominates the measures of accountability, 
guarantees that vocational education would become less and less concerned with the 
future and work change and thereby less and less able to develop new criteria and 
programs responsive to changing conditions. 

The concept of amployment in the field for which one is trained needs to be redefined 
to one that is more relevant to today's vvorkplar^ and to the individuars need to be able 
to adjust to the unknown changes in the workplace that will continue to come in the 
future. In fact, many employers argue that the greatest weakness of vocational 
education Is adherence too closely to specificity of skills which can become obsolete or 
prevent occupational mobility either vertically or horizontally. 



Conclusions 

Through this paper, the author has attempted no more than to suggest that traditional 
criteria for accountability in education and in vocational education are not relevant to 
the future^ Measurement of outcomes against only traditional and validated criteria 
assigns vocational ^education to a decreasing role in preparing individuals for future 
work at a time when preparing for work and changes in work roles are necessary for 
everyone. 

The new and specific needs of special groups and the new and continuing needs in 
education apply to everyone today and not just to a percentage of the population. 
Another way of expressing the arguments for the conclusions reached by this writer is 
that success criteria in both work preparation and in general education which have been 
traditionally used are not adequate. Somehow, the two must be put together and 
'Changed if vocational education is to meet the needs of both learner and society. Such 
redesign will require changes in product, process and in insiitutions which are 
responsible for product and process. 
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Therefore, the foiiowing conclusions should be given consideration m selectina cntona 
for accountability. 



1. Quality in yocational education cannot be tied to the old normative measures 
designed to select students out of formal education or to select workers to 
proceed into the higher skilled occupations. 

2. Criteria selected mus* be relevant to the individual needs as well as to new 
societal problems different from the past. 

3. Vocational education methods and processes need to focus more directly on 
the relationships and learning opportunities in society that will bring education 
and work experience together in a cooperative effort. 

4< Search for specific criteria that can be proven "to be right" may prevent the 
search for criteria that may be more usable idealistically and realistically. 

5. Criteria for accounlability must be selected that are responsive^o student 
needs, societal problems, and employer concerns before the evaluation 
processes can be determined. 



Specific Criteria Suggestions 

This paper has made no attempt to be definitive in the presentation of specific criteria 
Other than the suggestions made earlier. Perhaps if the suggested broad areas for 
criteria of accountability make sense a national effort to define specific criteria by a 
gfoup made up of employers, parents, students, teachers, administrators, policy makers, 
and iegislatqrs could be chosen to define specific criterion in each broad area/This 
could be a basis for redesigning the criteria for accountability against which vocational 
education should be held accountable. 

This effort ought to be made by those in vocational education so the concern for quality 
by vocational educators is obvious to those who now see compliance and regulation as 
the route to forcing vocational education to move toward new criteria. 

The four broad categories of criteria to be jdied are the following: 

1, Instructional and program quality, 

2, Program relevance to individual and societal needs in relation to work, 

3. Program impact on organization, policy, support and usage oi * ocational 
education, 

4. Individual transition to and growth in the work world. 

Unless the vocational educator and the ©valuator of vocational education propose new 
and better criteria acceptable to society, even more time will be spent in meeting 
compliance requirements which are becoming ends instead of means. Therefore, this 
writer proposes that these four broad categories of criteria become the basis for 
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establishing specific criterion in each area that can assess the broad changes of 
vocational education in specific units educatinr students or administering programs. 
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